Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 801–900 of 1,012
Petke, J., Martínez, M., Kechagia, M., Aleti, A., & Sarro, F. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663776 · Cited by 15
“5 [5] dataset, due to the availability of labelled patches.”
Fault Localization via Fine-Tuning Large Language Models with Mutation Generated Stack Traces
Jambigi, N., Bogacz, B., Mueller, M., Bach, T., & Felderer, M. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988982 · Cited by 7
“In the bug benchmark Defects4J [11], only 3.33% of the crashes contain crash-triggering test cases [12], which further emphasizes an approach to localize faults based on stack traces.”
InfluentialpyMethods2Test: A Dataset of Python Tests Mapped to Focal Methods
Abdelmadjid, I., & Dyer, R. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00123 · Cited by 4
“Defects4J [20] provides a set of reproducible bugs for Java, with tests to express the buggy behavior.”
NIODebugger: A Novel Approach to Repair Non-Idempotent-Outcome Tests with LLM-Based Agent
Ke, K. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00226 · Cited by 14
“RepairAgent [36] is the first work to address the program repair challenge using an LLM-agent-based technique, applied to the De-fects4J [44] dataset.”
SBEST: Spectrum-based fault localization without fault-triggering tests
Rafi, M. N., Pacheco, L. B. S., Chen, A. R., Yang, J., & Chen, T.-H. P. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10761-8 · Cited by 8
“Defects4J (Just et al. , 2014b) is a benchmark and framework of real bugs used in many previous Software Engineering studies related to bug repair and fault localization (Chen et al. , 2021b; Li et al. , 2021; Lutellier et al. , 2020; Ye et al. , 2022).”
InfluentialNew Strategies for Selecting Reuse Candidates on Automated Program Repair
Tanikado, A., Yokoyama, H., Yamamoto, M., Sumi, S., Higo, Y., & Kusumoto, S. (2017) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2017.175 · Cited by 4
Rahman, M. R., Golagha, M., & Pretschner, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3195097 · Cited by 5
Poster: BugZoo – A Platform for Studying Software Bugs
Timperley, C. S., Stepney, S., & Goues, C. L. (2018) · Cited by 4
The effectiveness of context-based change application on automatic program repair
Kim, J., Kim, J., Lee, E., & Kim, S. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09770-1 · Cited by 4
A Practical, Principled Measure of Fuzzer Appeal: A Preliminary Study
Gavrilov, M., Dewey, K. T., Groce, A., Zamanzadeh, D. J., & Hardekopf, B. (2020) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs51102.2020.00071 · Cited by 9
Division by Zero: Threats and Effects in Spectrum-Based Fault Localization Formulas
Vince, D., Szatmári, A., Kiss, Á., & Beszédes, Á. (2022) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs57517.2022.00032 · Cited by 4
HasBugs - Handpicked Haskell Bugs
Applis, L., & Panichella, A. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00040 · Cited by 6
“…code and GitHub; 3) Multi-point fault-locations, test-patches and 3-stage-versions (faulty, tested & fixed); 4) Dockerfiles and pre-produced available images running builds Similar Datasets are Defects4J [9] for Java programs, and in particular, the scripts and support created around it later 1 [10].”
BackgroundAugmenting Automated Spectrum Based Fault Localization for Multiple Faults
Chatterjee, P., Campos, J., Abreu, R., & Roy, S. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/350 · Cited by 5
“We have performed our experiments on six java project repositories, Chart, Closure, Lang, Math, Mockito and Time from the DEFECTS4J benchmark suite [Just et al., 2014].”
MethodContextAug: model-domain failing test augmentation with contextual information
Zhang, Z., Xue, J., Yang, D., & Mao, X. (2023) · Frontiers of Computer Science · doi:10.1007/s11704-023-2521-2 · Cited by 4
Compsuite: A Dataset of Java Library Upgrade Incompatibility Issues
Xu, X., Zhu, C., & Li, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00127 · Cited by 4
“[12] introduced Defects4J, a database supplies actual bugs, fixed program versions, and corresponding test suites.”
BackgroundMaes-Bermejo, M., Serebrenik, A., Gallego, M., Gortázar, F., Robles, G., & Gonzalez-Barahona, J. M. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10479-z · Cited by 8
“We applied the tool on a well-known bug dataset of 835 bugs and their corresponding BFCs, Defects4J (Just et al. 2014).”
InfluentialSystematic Analysis of Learning-Based Software Fault Localization
Zou, Y., Li, H., Li, D., Zhao, M., & Chen, Z. (2024) · International Symposium on System Security, Safety, and Reliability (ISSSR) · doi:10.1109/isssr61934.2024.00068 · Cited by 5
Evaluating the Repair Ability of LLM Under Different Prompt Settings
Tian, X. (2024) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse62657.2024.00053 · Cited by 13
“To evaluate LLM’s different repair ability when it is provided with different project information, we choose Defects4J 2.0 [41] as our dataset, which has been widely used in the program repair community.”
MethodInfluentialMining Bug Repositories for Multi-Fault Programs
Callaghan, D., & Fischer, B. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00023 · Cited by 4
“Fault localization and program repair tools are typically evaluated over bug repositories such as Defects4J [30] or BugsInPy [46].”
Improving the Efficiency of Search-Based Auto Program Repair by Adequate Modification Point
Jang, Y., Phung, Q.-N., & Lee, E. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-19063-7_56 · Cited by 4
Making Better Use of Repair Templates in Automated Program Repair: A Multi-Objective Approach
Yuan, Y., & Banzhaf, W. (2020) · Genetic and evolutionary computation · doi:10.1007/978-3-030-39831-6_26 · Cited by 4
Revisiting Test Cases to Boost Generate-and-Validate Program Repair
Zhang, J., Liu, K., Kim, D., Li, L., Liu, Z., Klein, J., & Bissyandé, T. F. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00010 · Cited by 5
BEIRUT: Repository Mining for Defect Prediction
Elmishali, A., Sotto-Mayor, B., Roshanski, I., Sultan, A., & Kalech, M. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00018 · Cited by 4
Test Benchmarks: Which One Now and in Future?
Artho, C. V., Benali, A., & Ramler, R. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00044 · Cited by 5
A Fault Localization Method Based on Metrics Combination
Ajibode, A., Shu, T., Said, K. S., & Ding, Z. (2022) · Mathematics · doi:10.3390/math10142425 · Cited by 6
Evaluating Automatic Fault Localization Using Markov Processes
Henderson, T. A. D., Podgurski, A., & Kucuk, Y. (2019) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2019.00021 · Cited by 4
Investigating the Relationship Between Mutants and Real Faults with Respect to Mutated Code
Kim, M.-G., 김능회, & In, H. P. (2020) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819402050028x · Cited by 4
NFL: Neighbor-Based Fault Localization Technique
Vancsics, B. (2019) · IEEE International Workshop on Intelligent Bug Fixing (IBF) · doi:10.1109/ibf.2019.8665491 · Cited by 4
Exploring Microtask Crowdsourcing as a Means of Fault LocalizationPreprint
Adriano, C., & van der Hoek, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.03015 · Cited by 5
Empirical Evaluation of Mutation-based Test Prioritization TechniquesPreprint
Shin, D., Yoo, S., Papadakis, M., & Bae, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04631 · Cited by 4
Assessment of spectrum-based fault localization for practical use
de Souza, H. A. (2018) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.45.2018.tde-17082018-153806 · Cited by 6
DroidBugs: An Android Benchmark for Automated Program RepairPreprint
Azevedo, L., Dantas, A., & Camilo-Junior, C. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.07353 · Cited by 6
Human-in-the-Loop Fault Localisation Using Efficient Test Prioritisation of Generated Tests.Preprint
An, G., & Yoo, S. (2021) · arXiv (Cornell University) · Cited by 4
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3
Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study
Wang, Y., Pradel, M., & Liu, Z. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764576 · Cited by 98
“Then we extensively investigate the prevalence of plausible patches that exhibit behavioral discrepancies from ground truth patches, the patch difference patterns leading to behavioral discrepancies, and the correctness of plausible patches that exhibit behavioral discrepancies.”
Demystifying API misuses in deep learning applications
Yang, D., Liu, K., Lei, Y., Li, L., Xie, H., Liu, C., Wang, Z., Mao, X., & Bissyandé, T. F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10413-9 · Cited by 4
“The availability of a curated dataset could always create new momentum for the associated research areas (e.g., Defects4J Just et al. 2014 for general Java bugs, and Mubench Amann et al. 2018 for Java API misuses).”
InfluentialBoosting mutation-based fault localization by effectively generating Higher-Order Mutants
Wu, S., Yang, B., Chang, Z., Li, Z., Chen, X., & Liu, Y. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2024.107660 · Cited by 3
LEAM++: Learning for Selective Mutation Fault Construction
Tian, Z., Chen, J., Wang, D., Zhu, Q., Fan, X., & Zhang, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3725528 · Cited by 5
“Although LEAM ++ is general, we just evaluated its effectiveness on Java projects (the widely-used Defects4J benchmark [48]), which may not represent the subjects under other programming languages.”
Towards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection
Yan, Y., Duong, V., Shao, H., & Poshyvanyk, D. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00177 · Cited by 4
“We avoid using real bug/vulnerability datasets [94]–[96] due to limitations like the absence of paired comments, lack of support for Python or Java, introduction to a new dataset domain etc. .”
Simulink Mutation Testing using CodeBERT
Zhang, J. Y., Ghobari, D., Sabetzadeh, M., & Nejati, S. (2025) · doi:10.1109/ast66626.2025.00009 · Cited by 3
“However, unlike code, which benefits from open-source repositories for identifying real-world fault patterns [15], Simulink models lack extensive public repositories, hindering the identification of real-world faults.”
Boosting Redundancy-Based Automated Program Repair by Fine-Grained Pattern Mining
Jiang, J., Li, F., Zhao, Z., Ye, Z., Liu, M., Wang, B., Zhang, H., & Chen, J. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00018 · Cited by 5
“Listing 1 presents the patch code of Codec-3 from the Defects4J [41], where a line starting with “+” denotes newly added code while starting with “-” denotes deleted code.”
InfluentialAutomated Generation of Issue-Reproducing Tests by Combining LLMs and Search-Based Testing
Kitsios, K., Castelluccio, M., & Bacchelli, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00165 · Cited by 13
“Alternative landmark datasets, such as Defects4J [38], have been saturated by recent LLMs [45].”
Kalyani, P., Rao, C. P., Goparaju, B., Babu, K. K., & Kandimalla, P. C. R. (2025) · Scientific Reports · doi:10.1038/s41598-025-31851-z · Cited by 5
“The second experiment was conducted in the opposite direction: we trained on BugSwarm and tested on Defects4J.”
InfluentialThe Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models
Ruiz, F. V., Hort, M., & Moonen, L. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756966 · Cited by 16
“Benchmarks: The evaluation of the models is done through two APR benchmarks in Java: Defects4J [17] and HumanEval-Java [15].”
EfficientEdit: Accelerating Code Editing via Edit-Oriented Speculative Decoding
Wang, P., Zhang, L., Liu, F., Zhu, Y. S., Xu, W., Shi, L., Lian, X., Li, M., Shen, B., & Fu, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00215 · Cited by 12
“Empirical studies also reveal that code editing—which includes modifying [11], [12], refactoring [13], or debugging existing code [14]—constitutes a highly frequent activity during the lifecycle of a software project [15]–[17].”
Let the Code Speak: Incorporating Program Dynamic State for Better Method-Level Fault Localization
Qin, Y., Wang, S., Lin, B., Peng, X. L., Ouyang, S., Chen, L., & Mao, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00072 · Cited by 3
“We adopt the widely-used benchmark Defects4J [12] for evaluation.”
InfluentialCharacterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges
Nashid, N., Ding, D., Gallaba, K., Hassan, A. E., & Mesbah, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00137 · Cited by 15
“While Defects4J [51], Bugs.jar [69], BugsInPy [70], and LMDefects [41] include them, others offer limited coverage [71]–[73] or focus on single-hunk edits [74]–[76].”
Assessing model-driven mutation testing of Java bytecode
Ancona, F., Bockisch, C. M., Neufeld, D., & Taentzer, G. (2026) · Software & Systems Modeling · doi:10.1007/s10270-026-01398-2 · Cited by 3
An Extensive Study on Multi-Priority Algorithm in Test Case Prioritization and Reduction
Long-bo, L., Zhou, Y., Yuan, Y., & Wu, S. (2021) · Asia Service Sciences and Software Engineering Conference (ASSE) · doi:10.1145/3456126.3456135 · Cited by 3
New Ranking Formulas to Improve Spectrum Based Fault Localization Via Systematic Search
Sarhan, Q. I., Gergely, T., & Beszédes, Á. (2022) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw55395.2022.00059 · Cited by 3
Static Call Graph Combination to Simulate Dynamic Call Graph Behavior
Ságodi, Z., Pengő, E., Jász, J., Siket, I., & Ferenć, R. (2022) · IEEE Access · doi:10.1109/access.2022.3229182 · Cited by 5
Automatic Repair of Java Programs with Mixed Granularity and Variable Mapping
Cao, H., Cui, Z., Deng, M., Chu, Y., & Meng, Y. (2023) · Information Technology And Control · doi:10.5755/j01.itc.52.1.30715 · Cited by 3
When debugging encounters artificial intelligence: state of the art and open challenges
Song, Y., Xie, X., & Xu, B. (2024) · Science China Information Sciences · doi:10.1007/s11432-022-3803-9 · Cited by 6
Exploring the Potential of Pre-Trained Language Models of Code for Automated Program Repair
Hao, S., Shi, X., & Liu, H. (2024) · Electronics · doi:10.3390/electronics13071200 · Cited by 9
ReClues: Representing and indexing failures in parallel debugging with program variables
Song, Y., Zhang, X., Xie, X., Liu, Q., Gao, R., & Xing, C. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639098 · Cited by 4
“Defects4J is one of the most popular benchmarks in the current field of software testing and debugging, due to its realism and ease-to-use [35].”
BackgroundSoftware Fault Localization Based on Weighted Association Rule Mining and Complex Networks
Wu, W., Wang, S., & Liu, B. (2024) · Mathematics · doi:10.3390/math12132113 · Cited by 3
P3: A Dataset of Partial Program Fixes
Beyer, D., Grunske, L., Kettl, M., Lingsch-Rosenfeld, M., & Raselimo, M. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644889 · Cited by 3
“More widely used datasets include Defects4J [11], BugsInPy [22], Space [21], and the Siemens suite of small C programs.”
MethodPredictive Mutation-Based Fault Localization: Balancing Effectiveness and Cost
Xu, S., Wu, S., Li, Z., Fan, L., & Liu, Y. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00078 · Cited by 3
“0.0) [7].”
MethodInfluentialLu, Y., Ye, S., & Qi, L. (2025) · Applied Sciences · doi:10.3390/app15073632 · Cited by 8
Naziri, M. M. A., Singh, A. K., Wu, B., Qin, F., Dutta, S., & d’Amorim, M. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731739 · Cited by 4
“…repository (SIR) [5] which was one of the bug datasets containing 81 artificial faults in projects across multiple languages, Defects4J [9] which includes 357 real bugs across five large real-world Java projects, BugsInPy [23] containing 493 real bugs from 17 real-world Python…”
Lee, H., An, G., & Yoo, S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00020 · Cited by 12
“We evaluate M ETAMON using 9,482 pairs of Java methods and their documentation, taken from five open source projects in Defects4J [18] version 2.0.1: the dataset contains both consistent and inconsistent pairs, since we generate half of the pairs using the buggy versions.”
InfluentialGraph-Based Fault Localization
Vancsics, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-24305-0_28 · Cited by 3
PRF: a framework for building automatic program repair prototypes for JVM-based languages
Ghanbari, A., & Marcus, A. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417929 · Cited by 3
Sentiment Polarity and Bug Introduction
Romano, S., Caulo, M., Scanniello, G., Baldassarre, M. T., & Caivano, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64148-1_22 · Cited by 3
Towards Evidence-Based Testability Measurements
Guglielmo, L., Riboni, A., & Denaro, G. (2021) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier52604.2021.00024 · Cited by 3
SPICA: A Methodology for Reviewing and Analysing Fault Localisation Techniques
Zhang, X., & Jiang, M. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00039 · Cited by 4
Enhancing Lexical Representation of Test Coverage for Failure Clustering
Yoon, J., & Yoo, S. (2021) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew52652.2021.00052 · Cited by 3
Estimating the potential of program repair search spaces with commit analysis
Etemadi, K., Tarighat, N., Yadav, S., Martínez, M., & Monperrus, M. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111263 · Cited by 5
M3V: Multi-modal Multi-view Context Embedding for Repair Operator Prediction
Xu, X., Wang, X., & Xue, J. (2022) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo53902.2022.9741261 · Cited by 7
An Empirical Study on Effects of Code Visibility on Code Coverage of Software Testing
Ma, L., Zhang, C., Yu, B., & Satō, H. (2015) · doi:10.1109/ast.2015.23 · Cited by 3
A Robustness-Based Confidence Measure for Hybrid System Falsification
Takisaka, T., Zhang, Z., Arcaini, P., & Ichiro Hasuo, A. (2022) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2022.3201157 · Cited by 3
“For example, Hemmati [31] evaluated the efficiency of statement, branch, MC/DC coverage, loop, and all-control flow coverage on five projects extracted from Defects4J [32], a benchmark of real faults for Java.”
BackgroundImproving Mutation-Based Fault Localization via Mutant Categorization
Li, X., & Tadikonda, D. N. (2022) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2022-157 · Cited by 3
On subsumption relationships in data flow testing
Chaim, M. L., Baral, K., Offutt, J., Neto, M. C., & de Araujo, R. P. A. (2023) · Software Testing Verification and Reliability · doi:10.1002/stvr.1843 · Cited by 3
“For our study, we chose 17 programs from the Defects4J repository [19], plus machine learning programs Elki (https:// elki-project.”
MethodApplication of the Law of Minimum and Dissimilarity Analysis to Regression Test Case Prioritization
Ufuktepe, E., & Tuğlular, T. (2023) · IEEE Access · doi:10.1109/access.2023.3283212 · Cited by 3
Debugging Flaky Tests using Spectrum-based Fault Localization
Gruber, M., & Fraser, G. (2023) · doi:10.1109/ast58925.2023.00017 · Cited by 4
“[30] also studied the impact of flaky tests on automated program repair based on tests that have been annotated as flaky in the Defects4J defect benchmark [51], and also found flakiness to significantly decrease fault localization efficacy, with localization results decreasing for 90 out of 387 bugs and none improving…”
MethodInfluentialResponsibility in Context: On Applicability of Slicing in Semantic Regression Analysis
Badihi, S., Ahmed, K., Li, Y., & Rubin, J. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00057 · Cited by 3
“We started with the popular Defects4J benchmark [18], which consists of 395 regression failures from six Java projects.”
BackgroundMethodViolationTracker: Building Precise Histories for Static Analysis Violations
Yu, P., Wu, Y., Peng, X., Peng, J., Zhang, J., Xie, P., & Zhao, W. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00171 · Cited by 3
“The existing work on bug datasets can be broadly categorized into the real bug [46]–[51], which may make projects fail to pass some test case(s), and static analysis violations [14], [27], [36], [52], [53] which may not fail to pass.”
BackgroundTest Case Prioritization Based on Neural Network Classification with Artifacts Traceability
Rotaru, I.-C., & Veșcan, A. (2023) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew60602.2023.00015 · Cited by 3
“The approach was validated against 5 C programs from the Software Infrastructure Repository (SIR) [16] and 5 Java programs from the Defects4J database [17].”
MethodAutomata-Based Trace Analysis for Aiding Diagnosing GUI Testing Tools for Android
Ma, E., Huang, S., He, W., Su, T., Wang, J., Liu, H., Pu, G., & Su, Z. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616361 · Cited by 3
“In our community, benchmarking software testing tools against a set of representative, ground-truth bugs ( e.g. , Defects4J [33], Lava [16], Magma [29]) is the well-justified and widely-used approach to evaluating and improving the tools’ bug finding abilities [34].”
MethodOn the suitability of checked coverage and genetic parameter tuning in test suite reduction
Koitz, R., Sterner, T., Stracke, L., & Wotawa, F. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2656 · Cited by 8
Combining Error Guessing and Logical Reasoning for Software Fault Localization via Deep Learning
Wang, R., Fan, M., Yan, Y. P., & Jiang, S. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194024500219 · Cited by 3
LLM Fault Localisation within Evolutionary Computation Based Automated Program Repair
Murtaza, S. B., Mccoy, A., Ren, Z., Murphy, A., & Banzhaf, W. (2024) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3638530.3664174 · Cited by 7
Verification and Validation of Quantum Software
Fortunato, D., Jiménez-Navajas, L., Campos, J., & Abreu, R. (2024) · Springer eBooks · doi:10.1007/978-3-031-64136-7_5 · Cited by 8
230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers
Alshammari, A., Ammann, P., Hilton, M. L., & Bell, J. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00031 · Cited by 3
“While datasets of true failures do exist [26], [27], [28], [29], these datasets are typically intentionally constructed from tests that are not flaky (to make studying the defects easier).”
BackgroundEvolutionary Testing for Program Repair
Ruan, H., Nguyen, H. L., Shariffdeen, R., Noller, Y., & Roychoudhury, A. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00058 · Cited by 7
“For our evaluation we use a subset of subjects from the popular D EFECTS 4J [7] benchmark v1.”
BackgroundMethodFoster, M. S., Wild, C., Hierons, R. M., & Walkinshaw, N. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00023 · Cited by 7
“One focus of future work would be to replicate this study on curated sets of real faults, such as Defects4J [38].”
BackgroundSearch-Based Automated Program Repair: A Survey
Zhang, S., Li, D., Zhao, M., Li, H., & Wong, W. E. (2024) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c63300.2024.00063 · Cited by 4
Uncertainty Estimation of Automatic Software Debugging in Open-Source Projects Hosting Platform
Liang, H., Guo, S., Li, H., & Li, C. H. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3524511 · Cited by 4
“The effectiveness of D-Helper is validated through extensive experiments on the Defects4j benchmark dataset [17].”
Revisiting Unnaturalness for Automated Program Repair in the Era of Large Language Models
Yang, A. Z. H., Kolak, S., Hellendoorn, V. J., Martins, R., & Goues, C. L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00089 · Cited by 30
“We use the Defects4J [28] dataset as the basis of our experiments.”
BackgroundMethodPoster: Supporting JavaScript Experimentation with BugsJS
Vancsics, B., Gyimesi, P., Stocco, A., Mazinanian, D., Beszédes, Á., Ferenć, R., & Mesbah, A. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00046 · Cited by 5
Improving Automated Program Repair with Retrospective Fault Localization
Xu, T. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00066 · Cited by 4
Learning Likely Invariants to Explain Why a Program Fails
Pham, L. H., Sun, J., Thi, L. L. T., Wang, J., & Peng, X. (2017) · IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) · doi:10.1109/iceccs.2017.12 · Cited by 3
Substate Profiling for Enhanced Fault Detection and Localization: An Empirical Study
Assi, R. A., Masri, W., & Trad, C. (2020) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst46399.2020.00013 · Cited by 4
Multiplicative Weights Algorithms for Parallel Automated Software Repair
Renzullo, J., Weimer, W. R., & Forrest, S. (2021) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps49936.2021.00107 · Cited by 3
Detecting Real Faults in the Gson Library Through Search-Based Unit Test Generation
Gay, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_25 · Cited by 3
Designing and Development of Database Testing Tool
Kaur, T., & Sehra, S. K. (2015) · International Journal of Computer Applications · doi:10.5120/21334-4330 · Cited by 3
Classification of Randomly Generated Test Cases
Artho, C. V., & Ma, L. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.32 · Cited by 4
Progress on Approaches to Automatic Program Repair
Xuan, J., Zhi-Lei, R., Wang, Z., Xie, X., & Jiang, H. (2016) · Cited by 4
Towards API-specific automatic program repair
Nielebock, S. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115721 · Cited by 4
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.