Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 901–1,000 of 1,012
LoopFix: an approach to automatic repair of buggy loops
Wang, W., Meng, Z., Wang, Z., Liu, S., & Hao, J. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.06.076 · Cited by 3
Multipartite Based Test Case Prioritization Using Failure History
Rahman, M. A., Hasan, M. A., Shah, K., & Siddik, M. S. (2019) · International Journal of Advanced Science and Technology · doi:10.33832/ijast.2019.125.03 · Cited by 3
Progress on software crash research
Jia, X., Xuan, J., Yongfeng, G. U., Ping, M., & Jiang, H. (2019) · Scientia Sinica Informationis · doi:10.1360/n112019-00018 · Cited by 3
An Evolutionary System for Better Automatic Software Repair
Yuan, Y., & Banzhaf, W. (2020) · Genetic and evolutionary computation · doi:10.1007/978-3-030-39958-0_19 · Cited by 3
Every Mutation Should Be Rewarded: Boosting Fault Localization with Mutated Predicates
Xu, X., Zou, C., & Xue, J. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00028 · Cited by 3
Automated Repair of Java Programs with Random Search via Code Similarity
Cao, H., Liu, F., Shi, J., Chu, Y., & Deng, M. (2021) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c55045.2021.00075 · Cited by 3
Effective Spectrum Based Fault Localization Using Contextual Based Importance Weight
Sarhan, Q. I., & Beszédes, Á. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-14179-9_7 · Cited by 4
Leveraging Causal Inference for Explainable Automatic Program Repair
Wang, J., Si, S., Zhu, Z., Qu, X., Hong, Z., & Xiao, J. Z. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892168 · Cited by 6
DroidLeaks: Benchmarking Resource Leak Bugs for Android ApplicationsPreprint
Liu, Y., Wei, L., Xu, C., & Cheung, S.-C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08079 · Cited by 3
DevReplay: Automatic Repair with Editable Fix PatternPreprint
Ueda, Y., Ishio, T., Ihara, A., & Matsumoto, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.11040 · Cited by 6
Xie, R., Zeng, Z., Yu, Z., Gao, C., Zhang, S., & Ye, W. (2024) · arXiv · doi:10.48550/arxiv.2403.15747 · Cited by 3
MuRS: Mutant Ranking and Suppression using Identifier Templates
Chen, Z., Salawa, M., Vijayvergiya, M., Petrović, G., Ivanković, M., & Just, R. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613901 · Cited by 2
“Evaluated on 9 projects from the Defects4J benchmark [9], they concluded that TCAP can be predicted from program context and that TCAP-based mutant prioritization improves test completeness faster than the previous state-of-the-art, which is random prioritization.”
BackgroundSelf-citationWhen Automated Program Repair Meets Regression Testing—An Extensive Study on Two Million Patches
Lou, Y., Yang, J., Benton, S., Hao, D., Tan, L., Chen, Z., Zhang, L. H., & Zhang, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672450 · Cited by 13
A Method for Processing Static Analysis Alarms Based on Deep Learning
Tan, Y., & Tian, J. (2024) · Applied Sciences · doi:10.3390/app14135542 · Cited by 4
Benefits and Pitfalls of Token-Level SZZ: An Empirical Study on OSS Projects
Watanabe, H., Kondo, M., Choi, E., & Mizuno, O. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00084 · Cited by 2
“One example is the Defects4J dataset [11], which contains bugs gathered from real-world Java systems.”
BackgroundOn the Stability and Applicability of Deep Learning in Fault Localization
Csuvik, V., Aszmann, R., Beszédes, Á., Horváth, F., & Gyimóthy, T. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00062 · Cited by 2
“In FL research [40], Defects4J is a well-known, widely used collection of Java programs and curated bugs.”
BackgroundEnhancing the Efficiency of Automated Program Repair via Greybox Analysis
Kim, Y., Park, Y., Han, S., & Yi, J. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695602 · Cited by 4
“We have shown the efficacy of Gresino through extensive experiments on the Defects4J benchmark and eight APR tools.”
InfluentialAn Empirical Study of Automatic Program Repair Techniques for Injection Vulnerabilities
Zhu, T., Xu, T., Liu, K., Zhou, J., Hu, X., Xia, X., Zhang, T., & Lo, D. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00014 · Cited by 2
“Since the investigated approaches are all originally evaluated on a widely used benchmark Defects4J [17], we propose to compare the patches in InjectionVul4J and Defects4J to identify their differences.”
An Empirical Study on the Relationship between Defects and Source Code’s Unnaturalness
Jiang, Y., Liu, H., Liu, J., Zhang, Y., Ji, W., Zhong, H., & Zhang, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3718083 · Cited by 3
“The first bug repository is Defects4J [31].”
Pugh, T., Lopez, C. M., Peacock, S., Deng, L., Dehlinger, J., & Chakraborty, S. (2025) · Software Testing Verification and Reliability · doi:10.1002/stvr.70004 · Cited by 5
Evaluating the Predictive Power of Software Metrics for Fault Localization
Arab, I., Magel, K. I., & Akour, M. (2025) · Computers · doi:10.3390/computers14060222 · Cited by 3
“We evaluated our approach on the Defects4J benchmark [27], which is a widely used dataset in this domain.”
Scalable Language Agnostic Taint Tracking using Explicit Data Dependencies
Effendi, S. D. B., Pinho, X., Dreyer, A. M., & Yamaguchi, F. (2025) · State Of the Art in Java Program Analysis · doi:10.1145/3735544.3735586 · Cited by 2
“To address this shortcoming, we use Defects4j [16] and include a Python-inspired variant, namely BugsInPy [28].”
Zhang, P., Lu, Z., Wang, Y., Yang, Y., Zhou, Y., & Papadakis, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3573751 · Cited by 4
“2 illustrates the defect in Defects4J [11], [13] indexed as Cli-33.”
Towards Better Answers: Automated Stack Overflow Post Updating
Mai, Y., Gao, Z., Wang, H., Bi, T., Hu, X., Xia, X., & Sun, J. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00024 · Cited by 14
“Datasets for code maintenance, often derived from programming competitions like Marmoset [63], IntroClass [64], QuixBugs [65], Codeflaws [66], or open-source projects such as iBugs [67], Defects4j [68], BugsJS [69], are pivotal for bug fixing tasks [70], [71], [72].”
SCOPE: Hybrid optimization strategy for higher-order mutation-based fault localization
Liu, H., Li, Z., Kang, X., Wu, S., Paul, D., Chen, X., & Liu, Y. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107873 · Cited by 3
A Cascaded Pipeline for Self-Directed, Model-Agnostic Unit Test Generation via LLMs
Ni, C., Wang, X., Yin, X., Chen, L., & Ma, G. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00037 · Cited by 4
Enhancing Fault Localization via Causal Measurement Features
Li, Y., Cai, B., Yang, H., Cai, C., Qiu, Y., Chen, T., & Zhang, C. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228088 · Cited by 2
“We perform experimental evaluations on the Defects4J [17] dataset, a widely used fault localization dataset that contains real open-source programs with real bugs that occurred in their historical versions and corresponding test cases.”
InfluentialCatCoder : Repository-Level Code Generation with Relevant Code and Type Context
Pan, Z., Hu, X., Xia, X., & Yang, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3779217 · Cited by 25
“As the methods in the benchmark are mined by the CoderUJB authors from the Defects4J [27] dataset, it is convenient to evaluate the correctness of the generated code using the existing extensive unit test methods.”
Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair
Huang, K., Zhang, J., Xie, X., & Chen, C. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00100 · Cited by 28
“These methods are typically evaluated on benchmarks like SWE-bench [15], Defects4J [24], and HumanEval [25], which are not designed to address multimodal tasks involving both visual and textual elements [22].”
Empirically Evaluating the Use of Bytecode for Diversity-Based Test Case Prioritisation
Elgendy, I. T., Hierons, R. M., & McMinn, P. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756969 · Cited by 2
“We ran our experiments using Java projects from the Defects4J framework [17].”
Review of Large Language Model-Based Software Fault Localization Techniques
Tian, P., Liu, P., & Li, Y. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00021 · Cited by 2
“However, it is sensitive to the quality and diversity of the test suite [18], the coincidental correctness (i.e., tests that execute faulty code without failing) [19], the interference between faults [20], among others.”
Large Language Models for Software Fault Localization: a Survey
Chang, X., Li, D., & Wong, W. E. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00024 · Cited by 2
FailMapper: Automated Generation of Unit Tests Guided by Failure Scenarios
Dong, R., Deng, Z., Zhu, X., Du, X., liu, H., Wang, S., Wen, S., & Xiang, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00197 · Cited by 3
“We then examined the prevalence of these failure scenarios in 854 real-world bugs from the Defects4J benchmark [16] for contemporary Java software.”
InfluentialDefects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs
Wang, J., Xie, X., Hu, Q., Liu, S., Yu, J., Kong, J., & Li, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00029 · Cited by 10
“However, these LLM-based techniques are mainly evaluated on Defects4J [5], and it is not clear their effectiveness on C/C++ projects.”
DLBench: A Comprehensive Benchmark for SQL Translation with Large Language Models
Lin, L., Chen, H., Zhu, Q., Chen, L., Tang, L., & Wu, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00076 · Cited by 4
“Recent LLMs exhibit remarkable capabilities in software development [29] and are evaluated using diverse code-related benchmarks spanning code generation [46]–[52], defect detection [53]–[57], program repair [58]–[62], and code summarization [63]–[67].”
LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework
Lops, A., Narducci, F., Ragone, A., Trizio, M., & Bartolini, C. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00198 · Cited by 6
“Thanks to its modular architecture, the framework can seamlessly integrate with alternative benchmarks such as D EFECTS 4J [46] or any other curated corpus of Java projects.”
CAMUS: Context-Aware Neural Mutation Selection
Chen, M., Wang, B., Lin, Y., & Zhang, J. M. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00013 · Cited by 2
“0 [26], the widely-adopted benchmark suite containing 501 bugs from 8 real-world Java projects, and ConDefects benchmark [27], an independent dataset containing 246 bugs in Mutation Testing, MBTCP, and MBFL.”
InfluentialFURINA: A Multi-Agent Framework for Automated Patch Correctness Assessment
Li, Y., Ji, Y., Chen, X., Shen, Z., Ge, J., & Luo, B. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3811825 · Cited by 5
“CACHE-small is a patch dataset for Defects4J [29], a widely used benchmark in APR and APCA methods.”
Živković, T., & Zivkovic, M. Z. (2026) · Evolutionary Intelligence · doi:10.1007/s12065-026-01225-z · Cited by 6
“For example, Defects4J [79], one of the few publicly available benchmarks suitable for testing generation and prioritization, was just marginally mentioned by a single study.”
Unveiling the power of code pre-trained models in automated program repair: A systematic review
Zhan, S., Wang, X. J., Wei, D., Xiang, X., Lin, Y., Wu, H., & Lu, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113015 · Cited by 3
Boosting LLMs for Mutation Generation
Wang, B., Deng, M., Chen, M., Yang, C., Lin, Y., Harman, M., Papadakis, M., & Zhang, J. M. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808156 · Cited by 4
Du, H., Palepu, V. K., & Jones, J. A. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797107 · Cited by 2
“Of these, 9 subjects are either from the Defects4J real-bug database [41] or”
Nam, D., Kim, T., Ryu, D., & Baik, J. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808179 · Cited by 6
“Unlike prior JIT corpora (e.g., SZZ-derived [30, 33]) or static snapshot datasets (e.g., Devign [69], Big-Vul [13], Defects4J [29]) that define “clean” at a snapshot or immediate post-fix Class balance (defective/clean) was preserved within each split, and splitting was performed per repository to…”
Kang, S., Yun, S., Hong, J., Yoo, S., & An, G. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786874 · Cited by 2
“It is larger by orders of magnitude than commonly used open-source benchmarks: even the largest projects from the widely-used Defects4J [13], BugsInPy [30], or SWE-bench [10] benchmarks have less than a million LOC.”
RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing
Li, X., Ben-Israel, N. I., Raz, Y., Ahmed, B., Serebro, D., & Raux, A. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817463 · Cited by 5
“While traditional datasets like De-fects4J [12] established baselines for automated program repair, they lack the repository-scale context required for evaluating modern Large Language Models.”
Automated Quality Assessment of Metamodel Evolution: A Case Study in Java Mutation Testing
Ancona, F., Wieber, P., Bockisch, C. M., & Taentzer, G. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3830307 · Cited by 2
“To do so, we used the Defects4J inventory of Java projects [18] with known bugs.”
Improving Mutation-Based Fault Localization with Plausible-code Generating Mutation Operators
Jeon, J., & Hong, S. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678577 · Cited by 2
Bridging the Gap between Different Programming Paradigms in Coverage-based Fault Localization
Li, F., Wang, M., & Hao, D. (2022) · Asia-Pacific Symposium on Internetware · doi:10.1145/3545258.3545272 · Cited by 2
Figra: evaluating a larger search space for cardumen in automatic program repair
Fonseca, A., & Oliveira, M. F. (2022) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1145/3524459.3527345 · Cited by 4
Improving Weighted-SBFL by Blocking Spectrum
Yoshioka, H., Higo, Y., & Kusumoto, S. (2022) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam55253.2022.00036 · Cited by 2
A Software Multi-Fault Clustering Ensemble Technology
Zhang, M., Wang, S., Wu, W., Qiu, W., & Xie, W. (2022) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c57518.2022.00059 · Cited by 2
Evolving Software: Combining Online Learning with Mutation-Based Stochastic Search
Renzullo, J., Weimer, W. R., & Forrest, S. (2023) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3597617 · Cited by 2
“From the Defects4J benchmark [Just et al. 2014], we evaluated 10 randomly selected defects from four open source Java projects.”
BackgroundMethodResultInfluentialA Language-agnostic Framework for Mining Static Analysis Rules from Code Changes
Effendi, S. D. B., Çirişci, B., Mukherjee, R., Nguyen, H. A., & Tripp, O. (2023) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip58684.2023.00035 · Cited by 2
“Inspired by Defects4J [15], Widyasari et al. [16] and Gyimesi et al. [17] built large scale benchmarks of bug & fix pairs for Python and JavaScript, respectively.”
Evaluating the Trade-offs of Text-based Diversity in Test Prioritisation
Khojah, R., Chao, C. H., & Neto, F. G. D. O. (2023) · doi:10.1109/ast58925.2023.00021 · Cited by 3
“We use open source data from Defects4J 5 [29] that provides unit tests that detect isolated faults along with specific information regarding tests that trigger such faults.”
MethodGLAD: Neural Predicate Synthesis to Repair Omission Faults
Kang, S., & Yoo, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00087 · Cited by 4
Contribution-based Test Case Reduction Strategy for Mutation-based Fault Localization (S)
Wang, H., Yang, K., Zhao, X., Cui, Y., & Wang, W. (2023) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2023-180 · Cited by 2
Doctor Code: A Machine Learning-Based Approach to Program Repair
Moosavi, S., Vahidi-Asl, M., Haghighi, H., & Rezaalipour, M. (2023) · Scientia Iranica · doi:10.24200/sci.2023.54718.3884 · Cited by 2
“- Comparison with APR tools for Java programs Some of the famous tools that are evaluated using Defects4j, are presented in Table 13.”
BackgroundMethodInfluentialFault Localization Using TrustRank Algorithm
Fan, X., Wu, K., Zhang, S., Yu, L., Zheng, W., & Ge, Y. (2023) · Applied Sciences · doi:10.3390/app132212344 · Cited by 2
Triggering Modes in Spectrum-Based Multi-location Fault Localization
Dao, T. M., Meng, N., & Nguyen, T. H. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613884 · Cited by 3
“Defects4J contains 835 bugs from 17 projects [31].”
BackgroundMethodSourcererJBF: A Java Build Framework For Large-Scale Compilation
Misu, M. R. H., Achar, R., & Lopes, C. V. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3635710 · Cited by 3
“We found that 10 articles used an existing compiled dataset, where six of them utilized Defect4J [ 27 ].”
BackgroundLeveraging Execution Trace with ChatGPT: A Case Study on Automated Fault Diagnosis
Sakura, T., Soga, R., Kanuka, H., Shimari, K., & Ishio, T. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00052 · Cited by 3
FixGPT: A Novel Three-Tier Deep Learning Model for Automated Program Repair
Ye, W., Xia, J., Feng, S., Zhong, X., Yuan, S., & Guan, Z. (2023) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc59305.2023.00078 · Cited by 4
Optimizing Mutation-Based Fault Localization Through Contribution-Based Test Case Reduction
Wang, H., Yang, K., & Wu, T. L. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819402450027x · Cited by 2
CrashJS: A NodeJS Benchmark for Automated Crash Reproduction
Oliver, P., Dietrich, J. B., Anslow, C., & Homer, M. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644912 · Cited by 2
“Aside from these new datasets, several older, more established datasets include the Qualitas Corpus [38], Da Capo [6], the Software-artifact Infrastructure Repository [14], and Defects4J [19], among others.”
BackgroundBenchmarking and Categorizing the Performance of Neural Program Repair Systems for Java
Zhong, W., Li, C., Liu, K., Ge, J., Luo, B., Bissyandé, T. F., & Ng, V. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3688834 · Cited by 5
“For empirical validation, we collect 411 bugs (119 bugs from Bears [45], 260 bugs from Defects4J [31], and 32 bugs from QuixBugs [38]) with corresponding test suites in the empirical part.”
InfluentialFeng, S., Yuan, S., Guan, Z., & Du, X. (2024) · Intelligent and Converged Networks · doi:10.23919/icn.2024.0016 · Cited by 2
The Impact of Program Reduction on Automated Program Repair
Vidziunas, L., Binkley, D., & Moonen, L. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00039 · Cited by 7
“We use Defects4J to illustrate the effectiveness of our approach on several real applications and to facilitate comparison with previous work [7].”
InfluentialAdversarial patch generation for automated program repair
Alhefdhi, A., Dam, H. K., Le-Cong, T., Le, B., & Ghose, A. (2025) · Software Quality Journal · doi:10.1007/s11219-025-09709-4 · Cited by 8
A Taxonomy of Real Faults for Hybrid Quantum-Classical Software Architectures
Bensoussan, A., Jahangirova, G., & Mousavi, M. R. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3788677 · Cited by 6
Leveraging Search-Based and Pre-Trained Code Language Models for Automated Program Repair
Lijzenga, O., Moghadam, I. H., & Zaytsev, V. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707774 · Cited by 5
“We conducted experiments on a subset of Defects4J [28], which may not cover the actual distribution of real-world bugs.”
InfluentialCan Test Generation and Program Repair Inform Automated Assessment of Programming Projects?
Gu, R., Rojas, J. M., & Shin, D. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988955 · Cited by 2
“First, compared to gPR defect benchmarks (e.g., Defects4J [26]) student submissions for programming assignments are relatively small, providing limited repair ingredients 4 .”
COSMosFL: Ensemble of Small Language Models for Fault Localisation
Cho, H., Kang, S., An, G., & Yoo, S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00007 · Cited by 6
“Our evaluation of COSMosFL using Defects4J [18] shows that ensembles can indeed outperform individual models when the FL task is constrained by energy consumption or token count.”
Towards Understanding Fine-Grained Programming Mistakes and Fixing Patterns in Data Science
Chen, W., Cheoh, J. L., Keim, M., Brunswicker, S., & Zhang, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729352 · Cited by 6
“For example, Defects4J [32] contains 395 Java bugs, Bugs.jar [61] includes 1,158 Java bugs along with their fixes, ManyBugs dataset [43] holds 185 C language bugs, and BugsInPy [80] documents 493 bugs in Python programs.”
An Empirical Study of Code Clones from Commercial AI Code Generators
Wu, W., Hu, H., Fan, Z., Qiao, Y., Huang, Y., Li, Y., Zheng, Z., & Lyu, M. R.-T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729397 · Cited by 11
“By contrast, studies on the data leakage issue focus on exploring the models’ generalization capability when there is no overlap between the models’ training and test sets [32, 61], meaning that the training and test tasks are totally different.”
The Sustainability Face of Automated Program Repair Tools
Martínez, M., Martínez‐Fernández, S., & Franch, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3744900 · Cited by 5
CodeDiffuSe: A masked diffusion framework for structure-aware code completion and repair
Onan, A., & Alhumyani, H. A. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00237-6 · Cited by 2
“The removal of semantic regularization leads to noticeable declines in functional metrics, such as test case accuracy on QuixBugs (5.4 points) and compilation rate on Defects4J (11.0 points).”
InfluentialPeer-aided repairer: empowering large language models to repair advanced student assignments
Zhao, Q., Zhang, L., Liu, F., Liu, Y., Yan, Z., Chen, Z., Zhou, Y., Jiang, J., Li, G., Sun, Z., et al. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10716-z · Cited by 10
“They evaluated both their approach on the Defects4J [22] and QuixBugs [27] dataset, and the results demonstrated their approach obtains the superior result on both datasets.”
Ufuktepe, E., Ufuktepe, D. K., & Karabulut, K. (2021) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac51774.2021.00129 · Cited by 2
Refining Fitness Functions for Search-Based Automated Program Repair
Guizzo, G., Blot, A., Callan, J., Petke, J., & Sarro, F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88106-1_12 · Cited by 2
CSFL: Fault Localization on Real Software Bugs Based on the Combination of Context and Spectrum
Yan, Y. P., Jiang, S., Zhang, S., & Huang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91265-9_12 · Cited by 2
Disposable Testing: Avoiding Maintenance of Generated Unit Tests by Throwing Them Away
Shamshiri, S., Campos, J., Fraser, G., & McMinn, P. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-c.2017.100 · Cited by 2
Learning Without Peeking: Secure Multi-party Computation Genetic Programming
Kim, J.-H., Epitropakis, M. G., & Yoo, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_13 · Cited by 2
Mapping Class Dependencies for Fun and Profit
Kanapala, A., & Gay, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_21 · Cited by 2
[Journal First] Do Automated Program Repair Techniques Repair Hard and Important Bugs?
Motwani, M., Sankaranarayanan, S., Just, R., & Brun, Y. (2018) · Cited by 2
Self-citationBytecode-Based Multiple Condition Coverage: An Initial Investigation
Bollina, S., & Gay, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59762-7_16 · Cited by 2
Leveraging Contextual Information from Function Call Chains to Improve Fault Localization
Beszédes, Á., Horváth, F., Penta, M. D., & Gyimóthy, T. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00122 · Cited by 2
Multi-Location Program Repair Strategies Learned from Successful Experience (S)
Wang, S., Mao, X., Niu, N., Yi, X., & Guo, A. (2019) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2019-007 · Cited by 6
Exploring the Characteristics of Spectra Distribution and Their Impacts on Fault Localization
Zhang, X., & Zheng, Z. (2020) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3383219.3383230 · Cited by 2
A Fault Localization and Debugging Support Framework driven by Bug Tracking Data
Hirsch, T. (2020) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew51248.2020.00053 · Cited by 2
Toward More Efficient Statistical Debugging with Abstraction Refinement
Zuo, Z., Niu, X., Zhang, S., Fang, L., Khoo, S., Lu, S., Sun, C. P., & Xu, G. H. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3544790 · Cited by 2
Digging into Semantics: Where Do Search-Based Software Repair Methods Search?
Ahmad, H., Cashin, P., Forrest, S., & Weimer, W. R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14721-0_1 · Cited by 2
Effectively Combining Risk Evaluation Metrics for Precise Fault Localization
Ajibode, A., Shu, T., Laghari, G., & Ding, Z. (2022) · Mathematics · doi:10.3390/math10213924 · Cited by 2
The promise and perils of using machine learning when engineering software (keynote paper)
Brun, Y. (2022) · doi:10.1145/3549034.3570200 · Cited by 2
Interactive Patch Filtering via Test Generation
Zhang, Q., Zhai, X., Xu, S., Huang, W., Zhang, J., & Fan, Y. (2022) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa56465.2022.00015 · Cited by 4
Are Neural Bug Detectors Comparable to Software Developers on Variable Misuse Bugs?
Richter, C., Haltermann, J., Jakobs, M.-C., Pauck, F., Schott, S., & Wehrheim, H. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3561156 · Cited by 6
Towards Context-Aware Spectrum-Based Fault Localization
Szatmári, A. (2023) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst57152.2023.00060 · Cited by 2
“We used the Defects4J [19] dataset to compare our proposed approach with the classical SBFL approach.”
MethodWang, B., Liu, G., Lin, Y., Ren, S., Li, H., & Zhang, D. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2624 · Cited by 2
Microtasking Software Failure Resolution
Adriano, C. (2019) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3310013.3310016 · Cited by 2
Software-Testing Contests: Observations and Lessons Learned
Wang, X., Sun, W., Hu, L., Zhao, Y. F., Wong, W. E., & Chen, Z. (2019) · Computer · doi:10.1109/mc.2019.2905533 · Cited by 2
How Compact Will My System Be? A Fully-Automated Way to Calculate LoC Reduced by Clone Refactoring
Nakagawa, T., Higo, Y., Matsumoto, J., & Kusumoto, S. (2019) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec48747.2019.00046 · Cited by 2
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.