Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 701–800 of 1,012
SBFL fault localization considering fault-proneness
Torkashvan, R., Parsa, S. A., & Vaziri, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112363 · Cited by 6
Impact of Large Language Models of Code on Fault Localization
Ji, S., Lee, S.-H., Lee, C.-S., Han, Y.-S., & Im, H. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989036 · Cited by 20
“To evaluate the FL performance of LLMCs fine-tuned using our approach and to compare it with previous studies, we mainly use the Defects4J benchmark [35] containing real defects from popular open-source Java projects.”
A large-scale study on human-cloned changes for automated program repair
Madeiral, F., & Durieux, T. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00064 · Cited by 6
Good Bug Hunting: Inspiring and Motivating Software Testing Novices
Silvis-Cividjian, N., Went, M., Jansma, R., Bonev, V., & Apostolov, E. (2021) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3430665.3456330 · Cited by 8
The ratio of equivalent mutants: A key to analyzing mutation equivalence
Marsit, I., Ayad, A., Kim, D., Latif, M., Loh, J., Omri, M. N., & Mili, A. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111039 · Cited by 6
A regression test case prioritization technique targeting ‘hard to detect’ faults
Biswas, S., Rathi, R., Dutta, A., Mitra, P., & Mall, R. (2021) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-021-01385-4 · Cited by 6
Yang, D., Lei, Y., Mao, X., Qi, Y., & Yi, X. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3561382 · Cited by 7
Can Deep Learning Models Learn the Vulnerable Patterns for Vulnerability Detection?
Yan, G., Chen, S., Bail, Y., & Li, X. (2022) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac54236.2022.00142 · Cited by 10
CirFix: Automated Hardware Repair and its Real-World Applications
Santiesteban, P., Huang, Y., Weimer, W. R., & Ahmad, H. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3269899 · Cited by 10
“IV-A] [66, Sec.”
Energy Consumption of Automated Program Repair
Martínez, M., Martínez‐Fernández, S., & Franch, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3643114 · Cited by 9
Reducing False Positives of Static Bug Detectors Through Code Representation Learning
Yang, Y., Wen, M., Gao, X. Y., Zhang, Y., & Sun, H. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00075 · Cited by 10
“Besides, we verify the effectiveness of the trained CRL-based techniques on a ground truth dataset [52] collected from Defects4J [53].”
MethodTest-based patch clustering for automatically-generated patches assessment
Martínez, M., Kechagia, M., Perera, A., Petke, J., Sarro, F., & Aleti, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10503-2 · Cited by 14
Ju, X., Qian, J., Chen, Z., Zhao, C., & Qian, J. (2020) · IEEE Access · doi:10.1109/access.2020.3037235 · Cited by 6
A Survey on Automatic Bug Fixing
Cao, H., Meng, Y., Shi, J., Li, L., Liao, T., & Zhao, C. (2020) · International Symposium on System and Software Reliability (ISSSR) · doi:10.1109/isssr51244.2020.00029 · Cited by 10
High-Quality Automated Program Repair
Motwani, M. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion52605.2021.00134 · Cited by 11
Quantifying Faultiness: What Does It Mean to Have N Faults?
Khaireddine, B., & Mili, A. (2021) · IEEE/ACM International Conference on Formal Methods in Software Engineering (FormaliSE) · doi:10.1109/formalise52586.2021.00013 · Cited by 6
A Framework of Vulnerable Code Dataset Generation by Open-Source Injection
Zhang, S. (2021) · IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA) · doi:10.1109/icaica52286.2021.9497888 · Cited by 6
GrasP: Graph-to-Sequence Learning for Automated Program Repair
Tang, B., Li, B., Bo, L., Wu, X., Cao, S., & Sun, X. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00091 · Cited by 8
Augmenting Diffs With Runtime Information
Etemadi, K., Sharma, A., Madeiral, F., & Monperrus, M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3324258 · Cited by 6
Evaluating Explanations for Software Patches Generated by Large Language Models
Sobania, D., Geiger, A., Callan, J., Brownlee, A. E. I., Hanna, C., Moussa, R., Zamorano, M., Petke, J., & Sarro, F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48796-5_12 · Cited by 12
Mutation-Based Evolutionary Fault Localisation
de Freitas, D. M., Leitao-Junior, P. S., Camilo-Junior, C. G., & Harrison, R. (2018) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2018.8477719 · Cited by 6
Exploring the Triggering Modes of Spectrum-Based Fault Localization: An Industrial Case
Dao, T. M., Wang, M., & Meng, N. (2021) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst49551.2021.00052 · Cited by 8
Exception-Driven Fault Localization for Automated Program Repair
Ginelli, D., Riganelli, O., Micucci, D., & Mariani, L. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00070 · Cited by 6
Critical Review of BugSwarm for Fault Localization and Program RepairPreprint
Durieux, T., & Abreu, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09375 · Cited by 13
PyBugHive: A Comprehensive Database of Manually Validated, Reproducible Python Bugs
Antal, G., Vándor, N., Kolláth, I., Mosolygó, B., Hegedűs, P., & Ferenć, R. (2024) · IEEE Access · doi:10.1109/access.2024.3449106 · Cited by 12
GBSR: Graph-based suspiciousness refinement for improving fault localization
Li, Z., Li, M., Wu, S., Xu, S., Chen, X., & Liu, Y. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112189 · Cited by 7
ROSE: An IDE-Based Interactive Repair Framework for Debugging
Reiss, S. P., Wei, X., Yuan, J. P., & Xin, Q. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705306 · Cited by 7
“For over 90% of the real bugs in the Defects4J dataset [45], the test cases relevant to the defective behavior of the software represent future data and were only introduced after the bug was identified.”
RetypeR: Integrated Retrieval-based Automatic Program Repair for Python Type Errors
Hao, S., Shi, X., & Liu, H. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00028 · Cited by 7
Revisiting Defects4J for Fault Localization in Diverse Development Scenarios
Rafi, M. N., Chen, A. R., Chen, T.-H. P., & Wang, S. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00022 · Cited by 11
“To help facilitate software testing research, several bug benchmarks have been proposed, with Defects4J [20] being the most popular and widely used.”
A Framework for Automated Software Testing using Machine Learning and Artificial Intelligence
Nagila, A., Trivedi, N., Nagila, R., Trivedi, K., Bhardwaj, S., & Rani, J. (2025) · International Conference on Knowledge Engineering and Communication Systems (ICKECS) · doi:10.1109/ickecs65700.2025.11034863 · Cited by 5
Challenging Bug Prediction and Repair Models with Synthetic Bugs
Ibrahimzada, A. R., Chen, Y., Rong, R., & Jabbarvand, R. (2025) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam67354.2025.00021 · Cited by 32
“…of buggy and correct code is similar (Algorithm 2 Lines 11– 2) Fine-tuning on real-world bugs: To avoid any bias in our conclusion based on synthetic bugs, we also fine-tuned baseline models with real-world bugs from three datasets, namely Defects4J [24] 5 , B UG S WARM [45], and RegMiner [42].”
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 7
Do LLMs Generate Useful Test Oracles? An Empirical Study with an Unbiased Dataset
Molinelli, D., Grazia, L. D., Martin-Lopez, A., Ernst, M. D., & Pezzè, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00031 · Cited by 16
“The most popular benchmark, Defects4J [27], has been publicly available since 2014 and is likely included in LLM training data of approaches evaluated after 2020.”
InfluentialSelf-citationNavigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Models
Liao, D., Yin, X., Pan, S., Ni, C., Xing, Z., & Sun, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00063 · Cited by 11
“To evaluate the effectiveness of JUnitGenie, we use real-world Java projects from Defects4J [36], a widely adopted benchmark for automated testing research.”
Identifying Metamorphic Relations: A Data Mutation Directed Approach
Sun, C., Jin, H., Wu, S., Fu, A., Wang, Z., & Chan, W. K. (2022) · Software Practice and Experience · doi:10.1002/spe.3280 · Cited by 12
“For NumberUtil, we collected its real-life faults from an open-source project, 60 and four faulty versions were thereby used.”
MethodEnsuring Critical Properties of Test Oracles for Effective Bug Detection
Hossain, S. B. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3639791 · Cited by 6
Evaluating String Distance Metrics for Reducing Automatically Generated Test Suites
Elgendy, I. T., Hierons, R. M., & McMinn, P. (2024) · doi:10.1145/3644032.3644455 · Cited by 5
“We used two state-of-the-art tools, Randoop and EvoSuite, to automatically generate regression test suites using their default configurations on real-world Java applications from the Defects4J [26] framework.”
MethodDeep Multiple Assertions Generation
Wang, H., Xu, T., & Wang, B. (2024) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3650105.3652293 · Cited by 10
“Considering that the buggy focal methods and test methods collected in Defects4J are from real-world projects and widely researched by the community, the evaluation result derived from this dataset is overall convincing.”
MethodInfluentialScalable Similarity-Aware Test Suite Minimization with Reinforcement Learning
Gu, S., & Mesbah, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715008 · Cited by 7
“• A comprehensive empirical evaluation, encompassing real-world open-source programs from the Defects4J [24] benchmark.”
Evaluating the Generalizability of LLMs in Automated Program Repair
Li, F., Jiang, J., Sun, J., & Zhang, H. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00024 · Cited by 14
“Although these LLM-based APR methods have achieved remarkable results, they have primarily only been evaluated on well-known datasets such as Defects4J [19] and QuixBugs [20], which were proposed years ago.”
A Database of Existing Vulnerabilities to Enable Controlled Testing Studies
Rei, S., & Abreu, R. (2017) · International Journal of Secure Software Engineering · doi:10.4018/ijsse.2017070101 · Cited by 5
Visualizing Code Genealogy: How Code is Evolutionarily Fixed in Program Repair?
Tomida, Y., Higo, Y., Matsumoto, S., & Kusumoto, S. (2019) · IEEE Working Conference on Software Visualization (VISSOFT) · doi:10.1109/vissoft.2019.00011 · Cited by 5
Graph Representation for Data Flow Coverage
Neto, M. C., de Araujo, R. P. A., Chaim, M. L., & Offutt, J. (2021) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac51774.2021.00128 · Cited by 5
Providing Real-time Assistance for Repairing Runtime Exceptions using Stack Overflow Posts
Mahajan, S., & Prasad, M. R. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00030 · Cited by 6
Evolving Ranking-Based Failure Proximities for Better Clustering in Fault Isolation
Song, Y., Xie, X., Zhang, X., Liu, Q., & Gao, R. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556922 · Cited by 5
RobustNPR: Evaluating the robustness of neural program repair models
Ge, H., Zhong, W., Li, C., Ge, J., Hu, H., & Luo, B. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2586 · Cited by 8
“However, due to the number of bugs provided by Defects4J, we could only use a limited number of samples for evaluation.”
BackgroundInfluentialSupConFL: Fault Localization with Supervised Contrastive Learning
Chen, W., Chen, W., Liu, J., Zhao, K., & Zhang, M. (2023) · Asia-Pacific Symposium on Internetware · doi:10.1145/3609437.3609441 · Cited by 6
“We use all the 6 Defects4J [16] V1.”
MethodImproving fault localization via weighted execution graph and graph attention network
Yan, Y. P., Jiang, S., Zhang, Y., & Zhang, C. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2619 · Cited by 7
Perera, A., Turhan, B., Aleti, A., & Böhme, M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3655022 · Cited by 11
“We use the Defects4J dataset (version 1.5.0) [28, 29] as our benchmark.”
InfluentialNL2Fix: Generating Functionally Correct Code Edits from Bug Descriptions
Fakhoury, S., Chakraborty, S., Musuvathi, M., & Lahiri, S. K. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3643526 · Cited by 11
MetaMFL: Metamorphic Multiple Fault Localization Without Test Oracles
Fu, L., Wu, Z., Lei, Y., & Yan, M. L. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3504400 · Cited by 6
Azadmanesh, M. R., & Hauswirth, M. (2015) · doi:10.1145/2807426.2807439 · Cited by 5
Automatic Reproducible Crash Detection
Gu, Y., Xuan, J., & Qian, T. (2016) · International Conference on Software Analysis, Testing and Evolution (SATE) · doi:10.1109/sate.2016.15 · Cited by 5
Evaluating Automated Program Repair Using Characteristics of Defects
Yokoyama, H., Higo, Y., & Kusumoto, S. (2017) · International Workshop on Empirical Software Engineering in Practice (IWESEP) · doi:10.1109/iwesep.2017.15 · Cited by 5
Test Case Prioritization Using Test Similarities
Haghighatkhah, A., Mäntylä, M., Oivo, M., & Kuvaja, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03673-7_18 · Cited by 6
Mining of Probabilistic Controlling Behavior Model From Dynamic Software Execution Trace
He, H., Ren, J., Zhao, G., Zhang, Y., & Hao, X. (2019) · IEEE Access · doi:10.1109/access.2019.2922998 · Cited by 5
Automatically Curated Data Sets
Kessel, M., & Atkinson, C. (2019) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2019.00015 · Cited by 9
A Dataset of Regressions in Web Applications Detected by End-to-End Tests
Soto-Sánchez, Ó., Maes-Bermejo, M., Gallego, M., & Gortázar, F. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-58793-2_35 · Cited by 5
Using defect prediction to improve the bug detection capability of search-based software testing
Perera, A. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3415286 · Cited by 7
Prioritized Test Generation Guided by Software Fault Prediction
Hershkovich, E., Stern, R., Abreu, R., & Elmishali, A. (2021) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw52544.2021.00045 · Cited by 8
The case for grammatical evolution in test generation
Murphy, A., Laurent, T., & Ventresque, A. (2022) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3520304.3534042 · Cited by 5
The improved dynamic slicing for spectrum-based fault localization
Cao, H., Wang, F., Deng, M., & Li, L. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1071 · Cited by 5
Wang, B., Liu, G., Lin, Y., Ren, S., Li, H., & Zhang, D. (2022) · Asia-Pacific Symposium on Internetware · doi:10.1145/3545258.3545286 · Cited by 5
didiffff: a viewer for comparing changes in both code and execution traces
Kanda, T., Shimari, K., & Inoue, K. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527877 · Cited by 7
Generated Tests in the Context of Maintenance Tasks: A Series of Empirical Studies
Herculano, W. B. R., Alves, E. L. G., & Mongiovi, M. (2022) · IEEE Access · doi:10.1109/access.2022.3222803 · Cited by 6
String Test Data Generation for Java Programs
Wang, M., Cui, B., Yan, J., Yan, J., & Zhang, J. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00033 · Cited by 5
Nilizadeh, A., Leavens, G. T., Păsăreanu, C. S., & Noller, Y. (2024) · Formal Aspects of Computing · doi:10.1145/3607538 · Cited by 10
“In this study, we did not use a dataset of real correct or buggy programs like Defects4J [68, 91]; however, Defects4J and other datasets do not have formal specifications.”
MethodUnit tests and component tests do make a difference on fault localisation effectiveness
Laghari, G., & Demeyer, S. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3194970 · Cited by 5
Substate Profiling for Effective Test Suite Reduction
Assi, R. A., Masri, W., & Trad, C. (2018) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2018.00023 · Cited by 5
Khalilian, A., Baraani-Dastjerdi, A., & Zamani, B. (2019) · Journal of Computer Languages · doi:10.1016/j.cola.2019.100927 · Cited by 5
Automated regression unit test generation for program merges
Ji, T., Chen, L., Mao, X., Yi, X., & Jiang, J. (2022) · Science China Information Sciences · doi:10.1007/s11432-019-3020-4 · Cited by 7
An empirical study of Linespots: A novel past‐fault algorithm
Scholz, M., & Torkar, R. (2021) · Software Testing Verification and Reliability · doi:10.1002/stvr.1787 · Cited by 7
The Relation between Bug Fix Change Patterns and Change Impact Analysis
Ufuktepe, E., Tuğlular, T., & Palaniappan, K. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00117 · Cited by 6
Gay, G. (2018) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1145/3194718.3194719 · Cited by 5
Improving Patch Quality by Enhancing Key Components of Automatic Program Repair
Soto, M. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00147 · Cited by 9
Revisiting ssFix for Better Program RepairPreprint
Qi, X., & Reiss, S. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.04583 · Cited by 6
E-APR: Mapping the Effectiveness of Automated Program RepairPreprint
Aleti, A., & Martínez, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03968 · Cited by 5
Megadiff: A Dataset of 600k Java Source Code Changes Categorized by Diff SizePreprint
Monperrus, M., Martínez, M., Ye, H., Madeiral, F., Durieux, T., & Yu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04631 · Cited by 25
Data Science and Machine Learning
(2022) · doi:10.5121/csit.2022.1215 · Cited by 5
Abstraction-Aware Inference of Metamorphic Relations
Nolasco, A., Molina, F., Degiovanni, R., Gorla, A., Garbervetsky, D., Papadakis, M., Uchitel, S., Aguirre, N., & Frias, M. F. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643747 · Cited by 13
“We inspected more than 200 classes from Defects4J [27], discarding classes with complex dependencies and with inheritance (Evo+EPA does not support them).”
MethodMutation Testing of Java Bytecode: A Model-Driven Approach
Bockisch, C. M., Eren, D., Lehmann, S., Neufeld, D., & Taentzer, G. (2024) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3640310.3674103 · Cited by 4
“All of our evaluations will use selected Java projects from the Defects4J benchmark [16].”
Learning-Based Automated Program Repair: A Systematic Literature Review
Chen, Q., Li, D., Zhao, M., Wong, W. E., & Li, H. (2025) · Complex System Modeling and Simulation · doi:10.23919/csms.2025.0004 · Cited by 5
Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation
Xu, Q., Wang, G., Briand, L., & Liu, K. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3803418 · Cited by 24
“But at this stage, we consider only Java methods that do not depend on user-defined classes or external libraries, thus excluding datasets such as Defects4J [45] and SF110 [29].”
State Field Coverage: A Metric for Oracle Quality
Molina, F., Aguirre, N., & Gorla, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00222 · Cited by 6
“The test assertions in our evaluation are taken from the Defects4J benchmark [26] (version 2.0.1), providing us with developer-written test assertions for real-world projects.”
InfluentialCan test cases generated by large language models facilitate automated program repair?
Zhang, C., Wang, H., Xu, C., Liu, J., Liu, K., & Liu, Z. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10802-w · Cited by 8
“We began by select - ing 374 single-function bugs from Defects4J (Just et al. 2014 ) and removing their corresponding trigger tests from the test suite.”
InfluentialEnhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review
Zhang, J., Hu, X., Gao, C., Xia, X., & Li, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802827 · Cited by 11
Characterizing and Detecting Program Representation Faults of Static Analysis Frameworks
Zhang, H., Pei, Y., Liang, S., Xing, Z., & Tan, S. H. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680398 · Cited by 13
Spotting Code Mutation for Predictive Mutation Testing
Zhao, Y., Chen, Y., Sun, Z., Liang, Q., Wang, G., & Hao, D. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695491 · Cited by 7
“Mutation testing is widely used to measure the test adequacy of a project, and mutation analysis is recognized as one of the strongest test-adequacy criteria [2, 16, 39].”
InfluentialEnhancing Fault Localization in Industrial Software Systems via Contrastive Learning
Li, C., Li, H., Li, Z., Pan, M., & Li, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00009 · Cited by 7
“Although fault localization has evolved significantly over decades and achieved considerable success, current methods are primarily evaluated on datasets such as Defects4J [15] and unit tests, leaving their effectiveness in large-scale software and system testing yet to be determined.”
Empirical evaluation of conditional operators in GP based fault localization
Kang, D., Sohn, J., & Yoo, S. (2017) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3071178.3071263 · Cited by 4
With registered reports towards large scale data curation
Herbold, S. (2020) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3377816.3381721 · Cited by 4
Looking for the Needle in the Haystack: End-to-end Tests in Open Source Projects
Gortázar, F., Maes-Bermejo, M., Gallego, M., & Padilla, J. C. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-85347-1_4 · Cited by 4
Fault-Based Regression Test Case Prioritization
Biswas, S., Bansal, A., Mitra, P., & Mall, R. (2022) · IEEE Transactions on Reliability · doi:10.1109/tr.2022.3205483 · Cited by 4
Toward a theory of program repair
Khaireddine, B., Zakharchenko, A., Martínez, M., & Mili, A. (2023) · Acta Informatica · doi:10.1007/s00236-023-00438-4 · Cited by 5
Finding Reusable Machine Learning Components to Build Programming Language Processing Pipelines
Flynn, P. J., Vanderbruggen, T., Liao, C., Lin, P., Emani, M. K., & Shen, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36889-9_27 · Cited by 6
An Approach to Regression Testing Selection based on Code Changes and Smells
Mori, A., Paiva, A. C. R., & Souza, S. R. S. (2023) · Brazilian Symposium on Systematic and Automated Software Testing (SAST) · doi:10.1145/3624032.3624036 · Cited by 4
“It provides a comprehensive database of known faults and corresponding test suites for various projects [13].”
BackgroundMethodContrastive Coincidental Correctness Representation Learning
Li, M., Lei, Y., Xie, H., Wang, J., Liu, C., & Deng, Z. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00074 · Cited by 8
“To evaluate the e ff ectiveness of our proposed method ContraCC, we conduct large-scale experiments on the widely-used Defects4J [32] benchmark.”
MethodAPI-Knowledge Aware Search-Based Software Testing: Where, What, and How
Ren, X., Ye, X., Lin, Y., Xing, Z., Li, S., & Lyu, M. R.-T. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616269 · Cited by 6
Investigating Execution Trace Embedding for Test Case Prioritization
Jabbar, E., Hemmati, H., & FELDT, R. H. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00036 · Cited by 7
“In this study, we run our experiments on 10 open-source Java projects from the Defects4J project [9]: Commons-Lang (Lang), Joda-Time (Time), Commons-Cli (Cli), JFreeChart (Chart), Commons-Compress (Compress), Jsoup, Mockito, jackson-databind (JacksonDatabind), jackson-core (Jackson-Core), and…”
MethodAljohani, A., & Do, H. (2024) · ACM Symposium on Applied Computing · doi:10.1145/3605098.3636058 · Cited by 12
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.