Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 301–400 of 1,012
On the Real-World Effectiveness of Static Bug Detectors at Finding Null Pointer Exceptions
Tomassi, D. A., & Rubio-González, C. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678535 · Cited by 26
Bui, Q.-C., Paramitha, R., Vu, D., Massacci, F., & Scandariato, R. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10415-7 · Cited by 50
“To compare the performance of the APR tools on fixing vulnerabilities vis-`a-vis fixing general bugs, we extracted the patch correctness percentage in the Defects4J dataset [30] from [47].”
InfluentialAre 20% of files responsible for 80% of defects?
Walkinshaw, N., & Minku, L. L. (2018) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3239235.3239244 · Cited by 33
VFL: Variable-based fault localization
Kim, J., Kim, J., & Lee, E. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.11.009 · Cited by 26
On the Effectiveness of Manual and Automatic Unit Test Generation: Ten Years Later
Serra, D., Grano, G., Palomba, F., Ferrucci, F., Gall, H. C., & Bacchelli, A. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00028 · Cited by 28
Demystifying Faulty Code: Step-by-Step Reasoning for Explainable Fault Localization
Widyasari, R., Ang, J. W., Nguyen, T. G., Sharma, N., & Lo, D. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00064 · Cited by 52
“This is in contrast to other fault localization datasets such as Defects4J [39], BugsJs [40], and BugsInPy [41], which are a collection of faulty Java, Javascript, and Python code, respectively.”
ResultMattis, T., Rein, P., Dürsch, F., & Hirschfeld, R. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387458 · Cited by 25
BUGSJS: a benchmark and taxonomy of JavaScript bugs
Gyimesi, P., Vancsics, B., Stocco, A., Mazinanian, D., Beszédes, Á., Ferenć, R., & Mesbah, A. (2020) · Software Testing Verification and Reliability · doi:10.1002/stvr.1751 · Cited by 36
Defect prediction guided search-based software testing
Perera, A., Aleti, A., Böhme, M., & Turhan, B. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416612 · Cited by 30
L2S: A Framework for Synthesizing the Most Probable Program under a Specification
Xiong, Y., & Wang, B. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487570 · Cited by 34
Widyasari, R., Prana, G. A. A., Haryono, S. A., Wang, S., & Lo, D. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10189-4 · Cited by 26
TransplantFix: Graph Differencing-based Code Transplantation for Automated Program Repair
Yang, D., Mao, X., Chen, L., Xu, X., Lei, Y., Lo, D., & He, J. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556893 · Cited by 29
Invalidator: Automated Patch Correctness Assessment Via Semantic and Syntactic Reasoning
Le-Cong, T., Luong, D. M., Le, X., Lo, D., Tran, N.-H., Bui, Q., & Huynh, Q.-T. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3255177 · Cited by 51
Detecting Condition-Related Bugs with Control Flow Graph Neural Network
Zhang, J., Wang, X., Zhang, H., Sun, H., Liu, X., Hu, C., & Liu, Y. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598142 · Cited by 37
“In this RQ, we further evaluate our approach by collecting real-world condition-related bugs from six unseen projects in the widely-used benchmark Defects4J [22].”
MethodDelta-Bench: Differential Benchmark for Static Analysis Security Testing Tools
Николаевич, П. И., Dashevskyi, S., & Massacci, F. (2017) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2017.24 · Cited by 27
Botsing, a search-based crash reproduction framework for Java
Derakhshanfar, P., Devroey, X., Panichella, A., Zaidman, A., & Arie Van Deursen, A. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3415299 · Cited by 25
Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs
Prenner, J. A., & Robbes, R. (2021) · arXiv · doi:10.48550/arxiv.2111.03922 · Cited by 25
“Extending this study to benchmarks that involve more complex codebases (e.g., Defects4J [21] or additional programming languages (e.g., BugsJS [22], a JavaScript benchmark) would provide additional interesting insights into Codex’ repair capabilities.”
Background(2020) · doi:10.1145/3379597.3387491 · Cited by 24
“specifically builttodetectSStuBs.ThescriptsusedtorunandevaluateSpotBugs are also available in our repository. 5 RELATED WORK Several previous data sets of real-world bugs have been curated. Defects4J [11] is a popular dataset consisting 395 Java bugs. Each bug is fixed in a single commit but the fix may modify multipl…”
BackgroundExploring Automated Assertion Generation via Large Language Models
Zhang, Q., Sun, W., Fang, C., Yu, B., Li, H., Yan, M. L., Zhou, J., & Chen, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3699598 · Cited by 47
“Defects4J [26] is a widely-used benchmark for evaluating software bug detection and automated program repair approaches.”
Automated Test Case Repair Using Language Models
Yaraghi, A. S., Holden, D., Kahani, N., & Briand, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3541166 · Cited by 57
“One of the more commonly used datasets is Defects4J [70], which contains hundreds of bugs from Java programs.”
An Empirical Study on API-Misuse Bugs in Open-Source C Programs
Gu, Z., Wu, J., Liu, J., Zhou, M. Q., & Gu, M. (2019) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2019.00012 · Cited by 24
Leveraging Contextual Information from Function Call Chains to Improve Fault Localization
Beszédes, Á., Horváth, F., Penta, M. D., & Gyimóthy, T. (2020) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054820 · Cited by 25
SMBFL: slice-based cost reduction of mutation-based fault localization
Chaleshtari, N. B., & Parsa, S. A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09845-4 · Cited by 24
Xiao, X., Pan, Y., Zhang, B., Hu, G., Li, Q., & Lu, R. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106653 · Cited by 24
Neural Program Repair : Systems, Challenges and Solutions
Zhong, W., Li, C., Ge, J., & Luo, B. (2022) · Asia-Pacific Symposium on Internetware · doi:10.1145/3545258.3545268 · Cited by 35
More Accurate Dynamic Slicing for Better Supporting Software Debugging
Li, X., & Orso, A. (2020) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst46399.2020.00014 · Cited by 25
Towards Better Graph Neural Network-Based Fault Localization through Enhanced Code Representation
Rafi, M. N., Kim, D. J., Chen, A. R., Chen, T.-H. (., & Wang, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660793 · Cited by 24
“0.0) benchmark [18], which contains 675 real-world faults from 14 open-source Java projects.”
InfluentialTest Case Prioritization Based on Dissimilarity Clustering Using Historical Data Analysis
Hasan, M. A., Rahman, M. A., & Siddik, M. S. (2017) · Communications in computer and information science · doi:10.1007/978-981-10-6544-6_25 · Cited by 23
Demystifying the Combination of Dynamic Slicing and Spectrum-based Fault Localization
Reis, S., Abreu, R., & d’Amorim, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/661 · Cited by 28
A Survey on the Use of Computer Vision to Improve Software Engineering Tasks
Bajammal, M., Stocco, A., Mazinanian, D., & Mesbah, A. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3032986 · Cited by 31
Islam, M. R., & Zibran, M. F. (2021) · ACM SIGAPP Applied Computing Review · doi:10.1145/3447332.3447334 · Cited by 23
Test case generation for agent-based models: A systematic literature review
Clark, A. G., Walkinshaw, N., & Hierons, R. M. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106567 · Cited by 33
A Large-scale Study on API Misuses in the Wild
Li, X., Jiang, J., Benton, S., Xiong, Y., & Zhang, L. (2021) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst49551.2021.00034 · Cited by 23
StandUp4NPR: Standardizing SetUp for Empirically Comparing Neural Program Repair Systems
Zhong, W., Ge, H., Ai, H., Li, C., Liu, K., Ge, J., & Luo, B. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556943 · Cited by 34
Bugs in machine learning-based systems: a faultload benchmark
Morovati, M. M., Nikanjam, A., Khomh, F., & Jiang, Z. M. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10291-1 · Cited by 52
Method-Level Bug Severity Prediction using Source Code Metrics and LLMs
Mashhadi, E., Ahmadvand, H., & Hemmati, H. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00055 · Cited by 43
“We use Defects4J [61] and Bugs.jar [62] datasets containing real bugs from different popular open-source Java projects.”
MethodGitBug-Java: A Reproducible Benchmark of Recent Java Bugs
Silva, A., Saavedra, N., & Monperrus, M. (2024) · Proceedings of the 21st International Conference on Mining Software Repositories · doi:10.1145/3643991.3644884 · Cited by 23
“Bug-fix benchmarks are integral to advancing the field of software engineering, providing crucial resources for soundly evaluating research in sub-fields such as automatic program repair (APR) and fault localization (FL) [8, 19, 22].”
BackgroundMethodInfluentialAn Improvement to Test Case Failure Prediction in the Context of Test Case Prioritization
Palma, F., Abdou, T., Bener, A., Maidens, J., & Liu, S. (2018) · doi:10.1145/3273934.3273944 · Cited by 23
Defexts: A Curated Dataset of Reproducible Real-World Bugs for Modern JVM Languages
Benton, S., Ghanbari, A., & Zhang, L. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00035 · Cited by 27
Multi-objective test suite optimization for detection and localization of software faults
Gupta, N., Sharma, A., & Pachariya, M. K. (2020) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2020.01.009 · Cited by 23
An Empirical Analysis of the Influence of Fault Space on Search-Based Automated Program RepairPreprint
Wen, M., Chen, J., Wu, R., Hao, D., & Cheung, S.-C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05172 · Cited by 31
Unit Test Case Generation with TransformersPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · Cited by 49
Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis
Li, F., Jiang, J., Sun, J., & Zhang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715004 · Cited by 96
“Listing 1 shows the developer patch and LLM’s patch for the bug JacksonDatabind-15 from Defects4J [42].”
Advancing Code Coverage: Incorporating Program Analysis with Large Language Models
Yang, C., Chen, J., Lin, B., Wang, Z., & Zhou, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748505 · Cited by 77
“To answer RQ5, we further evaluated TELPA on four real-world open-source projects using the JUnit framework [10] in the widely-used Defects4J benchmark [34] due to the significant popularity of Java and JUnit (i.e., Chart, Time, Lang, and Math).”
InfluentialIslam, M. R., & Zibran, M. F. (2020) · ACM Symposium on Applied Computing · doi:10.1145/3341105.3373880 · Cited by 22
Defects4J as a Challenge Case for the Search-Based Software Engineering Community
Gay, G., & Just, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59762-7_19 · Cited by 22
Self-citationTest case prioritization using test case diversification and fault-proneness estimations
Mahdieh, M., Mirian‐Hosseinabadi, S., & Mahdieh, M. (2022) · Automated Software Engineering · doi:10.1007/s10515-022-00344-y · Cited by 38
iBiR : Bug-report-driven Fault Injection
Khanfir, A., Koyuncu, A., Papadakis, M., Cordy, M., Bissyandé, T. F., Klein, J., & Traon, Y. L. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3542946 · Cited by 32
PyTER: effective program repair for Python type errors
Oh, W.-S., & Oh, H. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549130 · Cited by 22
Arcuri, A., Zhang, M., Golmohammadi, A., Belhadi, A., Galeotti, J. P., Mărculescu, B., & Seran, S. (2023) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst57152.2023.00047 · Cited by 22
“Another decade later, a very popular collection of Java programs with known, cataloged faults was Defect4J [41].”
BackgroundRueda, U., Just, R., Galeotti, J. P., & Vos, T. E. J. (2016) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1145/2897010.2897018 · Cited by 22
Self-citationBugs in the wild: examining the effectiveness of static analyzers at finding real-world bugs
Tomassi, D. A. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3275439 · Cited by 22
Ding, Z. Y., Lyu, Y., Timperley, C. S., & Goues, C. L. (2019) · IEEE/ACM International Workshop on Genetic Improvement (GI) · doi:10.1109/gi.2019.00011 · Cited by 23
Software Fault Localization Based on Network Spectrum and Graph Neural Network
Gou, X., Zhang, A., Wang, C., Liu, Y., Zhao, X., & Yang, S. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3374410 · Cited by 22
On the Impact of Flaky Tests in Automated Program Repair
Qin, Y., Wang, S., Liu, K., Mao, X., & Bissyandé, T. F. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00035 · Cited by 27
BugBuilder: An Automated Approach to Building Bug Repository
Jiang, Y., Liu, H., Luo, X., Zhu, Z., Chi, X., Niu, N., Zhang, Y., Hu, Y., Bian, P., & Zhang, L. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3177713 · Cited by 31
“A typical example is the well-known Defects4J [16].”
BackgroundMethodInfluentialWhen Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?
Chen, Y., Wu, J., Ling, X., Li, C., Rui, Z., Luo, T., & Wu, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3647633 · Cited by 52
“Existing datasets are either not built at repository level, such as QuixBugs [18], or cannot accurately restore scenarios of repository-level bugs, such as Defects4 [13].”
BackgroundEffectiveness and challenges in generating concurrent tests for thread-safe classes
Terragni, V., & Pezzè, M. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238224 · Cited by 23
Understanding Automatically-Generated Patches Through Symbolic Invariant Differences
Cashin, P., Martinez, C., Weimer, W. R., & Forrest, S. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00046 · Cited by 20
Tree2Tree Neural Translation Model for Learning Source Code Changes.Preprint
Chakraborty, S., Allamanis, M., & Ray, B. (2018) · arXiv (Cornell University) · Cited by 27
How Often Do Single-Statement Bugs Occur? The ManySStuBs4J DatasetPreprint
Karampatsis, R.-M., & Sutton, C. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13334 · Cited by 21
One Size Does Not Fit All: Multi-granularity Patch Generation for Better Automated Program Repair
Lin, B., Wang, S., Wen, M., Chen, L., & Mao, X. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680381 · Cited by 40
“In this section, we aim to evaluate the performance of Mulpor on Defects4J [29].”
Template-Guided Program Repair in the Era of Large Language Models
Huang, K., Zhang, J., Meng, X., & Liu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00030 · Cited by 54
Improving Spectrum-Based Localization of Multiple Faults by Iterative Test Suite Reduction
Callaghan, D., & Fischer, B. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598148 · Cited by 21
“We also contribute a new multi-fault dataset derived from the Defects4J dataset [30] of real bugs, and use it to evaluate FLITSR at the method and statement levels.”
BackgroundMethodOut of Context: How important is Local Context in Neural Program Repair?
Prenner, J. A., & Robbes, R. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639086 · Cited by 32
“Benchmarks such as Defects4J [8] or BugsInPy [32] come with test cases, which would alleviate this issue.”
MethodThe Fact Selection Problem in LLM-Based Program Repair
Parasaram, N., Yan, H., Yang, B., Flahy, Z., Qudsi, A., Ziaber, D., Barr, E. T., & Mechtaev, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00162 · Cited by 46
“Python Bug Benchmark: APR tools are typically compared on datasets of bugs extracted from real world projects, such as Defects4J [23] for Java and BugsInPy [10] for Python.”
Test case analytics: Mining test case traces to improve risk-driven testing
Noor, T. B., & Hemmati, H. (2015) · doi:10.1109/swan.2015.7070482 · Cited by 20
Single-objective Versus Multi-objectivized Optimization for Evolutionary Crash Reproduction
Soltani, M., Derakhshanfar, P., Panichella, A., Devroey, X., Zaidman, A., & Arie Van Deursen, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_18 · Cited by 21
An empirical assessment of baseline feature location techniques
Razzaq, A., Gear, A. L., Exton, C., & Buckley, J. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09734-5 · Cited by 25
Watch out for this commit! A study of influential software changes
Li, D., Li, L., Kim, D., Bissyandé, T. F., Lo, D., & Traon, Y. L. (2019) · Journal of Software Evolution and Process · doi:10.1002/smr.2181 · Cited by 25
Causal testing: understanding defects' root causesPreprint
Johnson, B., Brun, Y., & Meliou, A. (2020) · arXiv (Cornell University) · Cited by 21
FlexFL: Flexible and Effective Fault Localization With Open-Source Large Language Models
Xu, C.-Y., Liu, Z.-X., Ren, X.-X., Zhang, G., Liang, M., & Lo, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3553363 · Cited by 35
“Defects4J is a widely used benchmark for automatic fault localization, comprising a manually curated collection of real-world bugs from 17 Java projects [29], [53].”
Enriching automatic test case generation by extracting relevant test inputs from bug reports
Ouédraogo, W. C., Plein, L., Kaboré, K., Habib, A., Klein, J., Lo, D., & Bissyandé, T. F. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10635-z · Cited by 34
“Notably, EvoSuite could identify only about 23% of bugs from the Defects4J dataset [19].”
InfluentialDo Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?
Wang, G., Sun, Z., Ye, S., Gong, Z., Chen, Y., Zhao, Y., Liang, Q., & Hao, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771933 · Cited by 26
“In our evaluation of the code translation task, we compare three state-of-the-art approaches: Summarize-and-Generate (S&G) [3], Unitrans [56], and LostinTransl [34].”
InfluentialUnderstanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
Bouzenia, I., & Pradel, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00234 · Cited by 137
“For RepairAgent, we consider a trajectory successful if the produced patch passes the tests executed by Defects4J [32] (called “plausible”) and manual inspection confirms that the patch is equal or semantically to the developer-produced patch (called “correct”).”
Automated Customized Bug-Benchmark Generation
Kashyap, V., Ruchti, J., Kot, Ł., Turetsky, E., Swords, R., Pan, S. A., Henry, J., Melski, D., & Schulte, E. (2019) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2019.00020 · Cited by 19
Root cause prediction based on bug reports
Hirsch, T., & Hofer, B. (2020) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew51248.2020.00067 · Cited by 22
Research Progress of Flaky Tests
Zheng, W., Liu, G., Zhang, M., Chen, X., & Zhao, W. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00081 · Cited by 19
E-APR: Mapping the effectiveness of automated program repair techniques
Aleti, A., & Martínez, M. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09989-x · Cited by 32
Interactive Patch Filtering as Debugging Aid
Liang, J., Ji, R., Jiang, J., Zhou, S., Lou, Y., Xiong, Y., & Huang, G. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00028 · Cited by 27
Deep Learning Based Feature Envy Detection Boosted by Real-World Examples
Liu, B., Liu, H., Li, G., Niu, N., Xu, Z., Wang, Y., Xia, Y., Zhang, Y., & Jiang, Y. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616353 · Cited by 19
“The 5 projects in the second part were selected from Defects4J [31].”
BackgroundBLIP: Facilitating the Exploration of Undesirable Consequences of Digital Technologies
Pang, R. Y., Santy, S., Just, R., & Reinecke, K. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642054 · Cited by 30
“Learning from prior incidents has proven to be useful in several settings, ranging from exploratory forecasting of technological advances [60], improving software by studying a collection of previous defects [49], to training future pilots using the aviation accidents database [55].”
BackgroundSelf-citationGRT: An Automated Test Generator Using Orchestrated Program Analysis
Ma, L., Artho, C. V., Zhang, C., Satō, H., Gmeiner, J., & Ramler, R. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.102 · Cited by 22
Better Code Search and Reuse for Better Program Repair
Qi, X., & Reiss, S. P. (2019) · IEEE/ACM International Workshop on Genetic Improvement (GI) · doi:10.1109/gi.2019.00012 · Cited by 22
Experience Report: How Effective is Automated Program Repair for Industrial Software?
Noda, K., Nemoto, Y., Hotta, K., Hideo, T., & Kikuchi, S. (2020) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054829 · Cited by 19
FCCI: A fuzzy expert system for identifying coincidental correct test cases
Sabbaghi, A., Keyvanpour, M. R., & Parsa, S. A. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110635 · Cited by 19
Evaluating and Improving Unified Debugging
Benton, S., Li, X., Lou, Y., & Zhang, L. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3125203 · Cited by 20
Saying ‘Hi!’ is not enough: Mining inputs for effective test generation
Toffola, L. D., Staicu, C.-A., & Pradel, M. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115617 · Cited by 23
Evaluating data-flow coverage in spectrum-based fault localization
Ribeiro, H. L., de Araujo, P. A. R., Chaim, M. L., de Souza, H. A., & Kon, F. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870182 · Cited by 24
Leveraging Large Language Model for Automatic Patch Correctness Assessment
Zhou, X., Xu, B., Kim, K., Han, D., Nguyen, H. H., Le-Cong, T., He, J., Le, X.-B. D., & Lo, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3452252 · Cited by 43
“In contrast, our work mainly focuses on the Defects4J [61] and Bears [65] benchmarks with the use of LLMs.”
MethodSurvey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends
Lan, Y., Li, X., Du, H., Lu, X., Gao, M. Z., Qian, W., & Zhou, A. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3621181 · Cited by 61
“Defects4J [185], ManyBugs [186] and IntroClass [186] are some of the earlier datasets proposed for CEC tasks, focusing on Java and C languages respectively, but with small amounts of data.”
Slicer4J: a dynamic slicer for Java
Ahmed, K., Lis, M., & Rubin, J. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473123 · Cited by 33
FlakiMe: laboratory-controlled test flakiness impact assessment
Cordy, M., Rwemalika, R., Franci, A., Papadakis, M., & Harman, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510194 · Cited by 29
DeepMutation: a neural mutation tool
Tufano, M., Kimko, J., Wang, S., Watson, C., Bavota, G., Penta, M. D., & Poshyvanyk, D. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377812.3382146 · Cited by 38
How do Java mutation tools differ?
Amalfitano, D., Paiva, A. C. R., Inquel, A., Pinto, L. M. G., Fasolino, A. R., & Just, R. (2022) · Communications of the ACM · doi:10.1145/3526099 · Cited by 26
Self-citationXAI4FL: enhancing spectrum-based fault localization with explainable artificial intelligence
Widyasari, R., Prana, G. A. A., Haryono, S. A., Tian, Y., Zachiary, H. N., & Lo, D. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527902 · Cited by 26
Instance Space Analysis of Search-Based Software Testing
Neelofar, N., Smith‐Miles, K., Muñoz, M. A., & Aleti, A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3228334 · Cited by 28
“benchmark in automated testing literature [39], [45], [46], [47], [48], [49], [50] because they are usually free of complex environmental dependencies and can be tested easily without complex inputs.”
BackgroundInfluentialSyntactic Versus Semantic Similarity of Artificial and Real Faults in Mutation Testing Studies
Ojdanić, M., Garg, A., Khanfir, A., Degiovanni, R., Papadakis, M., & Traon, Y. L. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3277564 · Cited by 35
“2In this paper we use the term “real faults” to refer to the set of reproducible curated faults provided at the Defects4J dataset [29].”
MethodAutomated Program Repair, What Is It Good For? Not Absolutely Nothing!
Eladawy, H. M., Goues, C. L., & Brun, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639095 · Cited by 40
“We used the Defects4J dataset of real-world defects from open-source software systems that occurred during the natural development process [28].”
MethodInfluentialThe GitHub Recent Bugs Dataset for Evaluating LLM-Based Debugging Applications
Lee, J. Y., Kang, S., Yoon, J., & Yoo, S. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00049 · Cited by 41
“We further confirm that these bugs were not • We provide a partial evaluation into how pervasive • We provide an automated framework to gather real-world • We make our bug benchmark, GHRB, publicly available”
Background
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.