Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 501–600 of 1,012
Wang, X., Yu, H., Meng, X., Cao, H., Zhang, H., Sun, H., Liu, X., & Hu, C. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3654441 · Cited by 18
“Among them, TBar [42] is the state-of-the-art template-based repair technique, which contains 15 commonly used fix templates and achieves a good performance on the benchmark Defects4J [25].”
Fine-grained, accurate and scalable source differencing
Falleri, J., & Martínez, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639148 · Cited by 30
“The first two are Defects4J [22] (D4J) and BugsInPy [39] (BIP).”
BackgroundMethodGitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions
Saavedra, N., Silva, A., & Monperrus, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3640023 · Cited by 25
“For example, Defects4J [3] is a widely adopted bug-fix benchmark that has greatly served software engineering research in the past decade.”
BackgroundMethodAn empirical study of fault localization in Python programs
Rezaalipour, M., & Furia, C. A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10475-3 · Cited by 14
An Empirical Study on Focal Methods in Deep-Learning-Based Approaches for Assertion Generation
He, Y., Huang, J., Yu, H., & Xie, T. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660785 · Cited by 23
“We notice that the bug-finding metric has been used to evaluate TOGA [6, 15] on the Defects4J benchmark [12], and we plan to use this evaluation metric to evaluate all DLAGs in future work.”
Deep Learning Library Testing: Definition, Methods and Challenges
Zhang, X., Jiang, W., Shen, C. Q., Li, Q. L., Wang, Q., Lin, C., & Guan, X. (2025) · ACM Computing Surveys · doi:10.1145/3716497 · Cited by 33
“Researchers could refer to existing software testing datasets and benchmarks [6, 53, 66, 74] to build a dataset that covers a large number of APIs and various types of bugs and use virtual environment management tools like Docker, thus supporting bug datasets for multiple DL frameworks and releases.”
MORepair : Teaching LLMs to Repair Code via Multi-Objective Fine-Tuning
Yang, B., Tian, H., Ren, J., Zhang, H., Klein, J., Bissyandé, T. F., Goues, C. L., & Jin, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735129 · Cited by 67
“The derivative of a polynomial term 𝑎𝑥 is 𝑛𝑎𝑥 − 1 , meaning each coefficient must be multiplied by its corresponding power.”
Prioritizing Dissimilar Test Cases in Regression Testing using Historical Failure Data
Abdur, M., Abu, M. S., & Saeed, M. (2018) · International Journal of Computer Applications · doi:10.5120/ijca2018916258 · Cited by 12
EdSynth: Synthesizing API Sequences with Conditionals and Loops
Yang, Z., Hua, J., Wang, K., & Sarfraz Khurshid, D. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00025 · Cited by 15
Timperley, C. S., Stepney, S., & Goues, C. L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3195050 · Cited by 12
Riganelli, O., Micucci, D., & Mariani, L. (2018) · Software Practice and Experience · doi:10.1002/spe.2672 · Cited by 12
Is the Stack Distance Between Test Case and Method Correlated With Test Effectiveness?
Niedermayr, R., & Wagner, S. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319021 · Cited by 12
Efilter: An effective fault localization based on information entropy with unlabelled test cases
Xiaobo, Y., Liu, B., Wang, S., Dong, A., Feng, Z., & Yelin, Y. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106543 · Cited by 15
Revisiting the building of past snapshots — a replication and reproduction study
Maes-Bermejo, M., Gallego, M., Gortázar, F., Robles, G., & Gonzalez-Barahona, J. M. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10117-6 · Cited by 12
Evaluating and Improving Static Analysis Tools Via Differential Mutation Analysis
Groce, A., Ahmed, I., Feist, J., Grieco, G., Gesi, J., Meidani, M., & Chen, Q. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00032 · Cited by 13
Yan, L., Wen, T., Xie, H., Fu, L., Liu, C., Xu, L., & Sun, H. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00045 · Cited by 15
Evaluating the Impact of Experimental Assumptions in Automated Fault Localization
Soremekun, E., Kirschner, L., Böhme, M., & Papadakis, M. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00025 · Cited by 18
FixEval: Execution-based Evaluation of Program Fixes for Programming Problems
Haque, M. M. A., Ahmad, W. U., Lourentzou, I., & Brown, C. (2023) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr59189.2023.00009 · Cited by 46
Xiong, Y., Wang, B., Fu, G., & Zang, L. (2018) · doi:10.1145/3194810.3194816 · Cited by 16
Behavioral Fault Localization by Sampling Suspicious Dynamic Control Flow Subgraphs
Henderson, T. A. D., & Podgurski, A. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00019 · Cited by 12
Simulating the Effect of Test Flakiness on Fault Localization Effectiveness
Vancsics, B., Gergely, T., & Beszédes, Á. (2020) · doi:10.1109/vst50071.2020.9051636 · Cited by 14
Perera, A., Aleti, A., Turhan, B., & Böhme, M. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3147008 · Cited by 19
Generating Concise Patches for Newly Released Programming Assignments
Li, L., Liu, H., Li, K., Jiang, Y., & Sun, R. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3153522 · Cited by 17
Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Capability
Shin, D., Yoo, S., & Bae, D. (2016) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2016.37 · Cited by 12
Improving Retrieval-Augmented Deep Assertion Generation via Joint Training
Zhang, Q., Fang, C., Zheng, Y., Qian, R., Yu, S., Zhao, Y., Zhou, J., Yang, Y., Zheng, T., & Chen, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3545970 · Cited by 17
“Following prior work [1], [72], [73], we utilize EvoSuite [10] to generate test cases for Defects4J [74] that contains 835 bugs from 17 real-world Java projects.”
Effective Isolation of Fault-Correlated Variables via Statistical and Mutation Analysis
Wen, M., Xie, Z., Luo, K., Chen, X., Yang, Y., & Jin, H. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3209590 · Cited by 13
“To evaluate the effectiveness of I soVar, we applied it to real software bugs from the Defects4J [30] and Bears benchmark [31], which are widely adopted and evaluated in FL and APR.”
BackgroundMethodInfluentialCONCORD: Clone-Aware Contrastive Learning for Source Code
Ding, Y., Chakraborty, S., Buratti, L., Pujar, S., Morari, A., Kaiser, G. E., & Ray, B. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598035 · Cited by 26
“We design heuristics to generate such deviants based on the observations from a wide number of reported, real-world bug patterns [3, 12, 20, 21, 44, 46, 59, 60].”
BackgroundBUGSC++: A Highly Usable Real World Defect Benchmark for C/C++
An, G., Kwon, M., Choi, K., Yi, J., & Yoo, S. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00208 · Cited by 17
“0 [4] Java Yes (835, 17) BugsJS [16] JavaScript Yes (453, 10) FixJS [17] JavaScript Yes (323,907, -) BugsInPy [18] Python Yes (493, 17) Refactory [19] Python No † Defexts [20] Kotlin, Groovy, Scala Yes (654, 413) BugSwarm [21] Python, Java Yes (2,611, -) QuixBugs [22] Python, Java No ‡ † : Class…”
BackgroundMethodT5APR: Empowering automated program repair across languages through checkpoint ensemble
Gharibi, R., Sadreddini, M. H., & Fakhrahmad, S. M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112083 · Cited by 20
“Defects4J (v1.”
MethodInfluentialNeural-MBFL: Improving Mutation-Based Fault Localization by Neural Mutation
Du, B., Han, B., Liu, H., Chang, Z., Liu, Y., & Chen, X. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00168 · Cited by 14
“The Defects4J benckmark [28] is a valuable collection of repeatable faults found in major open-source applications, mainly designed to support software engineering research.”
BackgroundBenchmarking Automated Program Repair: An Extensive Study on Both Real-World and Artificial Bugs
Ouyang, Y., Yang, J., & Zhang, L. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652140 · Cited by 34
“To evaluate the APR techniques, many real-world bug datasets have been proposed, such as Bears [31], Bugs.jar [44], QuixBugs [22] and Defects4J [20].”
Patch Correctness Assessment: A Survey
Fei, Z., Ge, J., Li, C., Wang, T., Li, Y., Zhang, H., Huang, L., & Luo, B. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702972 · Cited by 25
Machine Learning Approaches to Code Similarity Measurement: A Systematic Review
Zhang, Z., & Saber, T. (2025) · IEEE Access · doi:10.1109/access.2025.3553392 · Cited by 25
Luo, W., Keung, J., Yang, B., Ye, H., Goues, C. L., Bissyandé, T. F., Tian, H., & Le, X.-B. D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3733599 · Cited by 36
“All methods and LLMs use the prompt structure uniformly to ensure consistency and mitigate potential biases.”
Paterson, D., Kapfhammer, G. M., Fraser, G., & McMinn, P. (2018) · International Workshop on Automation of Software Test (AST) · doi:10.1145/3194733.3194735 · Cited by 11
Leveraging Fault Localisation to Enhance Defect Prediction
Sohn, J., Kamei, Y., McIntosh, S., & Yoo, S. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00034 · Cited by 13
Program Repair With Repeated Learning
Chen, L., Pei, Y., Pan, M., Zhang, T. Y., Wang, Q., & Furia, C. A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3164662 · Cited by 18
Fine-Tuning GPT-2 to Patch Programs, Is It Worth It?
Lajkó, M., Horváth, D., Csuvik, V., & Vidács, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-10542-5_6 · Cited by 20
Tang, L., Bao, L., Xia, X., & Huang, Z. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00037 · Cited by 27
“They use the detailed information provided by the Defects4J dataset [40], including documents of changes in the version control system and patches with the exact changes to re-introduce the bug.”
MethodA Large-Scale Empirical Review of Patch Correctness Checking Approaches
Yang, J., Wang, Y., Lou, Y., Wen, M., & Zhang, L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616331 · Cited by 38
Tiwari, N. M., Upadhyaya, G., & Rajan, H. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2892662 · Cited by 11
Wong, C.-P., Meinicke, J., & Kästner, C. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264837 · Cited by 11
Automatically Patching Vulnerabilities of Binary Programs via Code Transfer From Correct Versions
Hu, Y., Zhang, Y., & Gu, D. (2019) · IEEE Access · doi:10.1109/access.2019.2901951 · Cited by 17
Why train-and-select when you can use them all? ensemble model for fault localisation
Sohn, J., & Yoo, S. (2019) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3321707.3321873 · Cited by 11
Search-based fault localisation: A systematic mapping study
Leitao-Junior, P. S., de Freitas, D. M., Vergílio, S. R., Camilo-Junior, C. G., & Harrison, R. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106295 · Cited by 11
Boosting automated program repair with bug-inducing commits
Wen, M., Liu, Y., & Cheung, S.-C. (2020) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3377816.3381743 · Cited by 11
Is the Ground Truth Really Accurate? Dataset Purification for Automated Program Repair
Yang, D., Lei, Y., Mao, X., Lo, D., Xie, H., & Yan, M. L. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00018 · Cited by 15
Efficiently Finding Data Flow Subsumptions
Chaim, M. L., Baral, K., Offutt, J., Neto, M. C., & de Araujo, R. P. A. (2021) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst49551.2021.00021 · Cited by 11
Reducing the search space of bug inducing commits using failure coverage
An, G., & Yoo, S. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473129 · Cited by 11
An Empirical Study on the Effect of Dynamic Slicing on Automated Program Repair Efficiency
Guo, A., Mao, X., Yang, D., & Wang, S. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00066 · Cited by 11
Delta4Ms: Improving mutation‐based fault localization by eliminating mutant bias
Liu, H., Li, Z., Han, B., Liu, Y., Chen, X., & Liu, Y. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1872 · Cited by 10
“In this subsection, we conduct a preliminary experiment to investigate the fault localization effectiveness generalization of Delta4Ms on the Defects4J Benchmark [68].”
MethodHetFL: Heterogeneous Graph-Based Software Fault Localization
Chen, X., Sun, T., Zhuang, D., Yu, D., Jiang, H., Zhou, Z., & Li, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3454605 · Cited by 13
Statically Contextualizing Large Language Models with Typed Holes
Blinn, A., Li, X., Kim, J., & Omar, C. (2024) · Proc. ACM Program. Lang. · doi:10.1145/3689728 · Cited by 25
“Repository-level benchmarks like RepoEval [83], RepoBench [42], CrossCodeEval [20], the Co-CoMIC dataset [19], and defects4j [34] are more suitable because they are high-context, i.e. they require completing code that depends on definitions in different files.”
Context-based Transfer Learning for Structuring Fault Localization and Program Repair Automation
Zhang, L., Guo, S., Guo, Y., Li, H., Chai, Y., Chen, R., Li, X., & Jiang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705302 · Cited by 12
Context-aware prompting for LLM-based program repair
Li, Y., Cai, M., Chen, J., Xu, Y., Huang, L., & Li, J. K. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00512-w · Cited by 24
“To evaluate the performance and generalizability of CodeCorrector, we employ the popular and widely-used benchmark in program repair, i.e., Defects4J(Just et al. 2014).”
Less Is More: On the Importance of Data Quality for Unit Test Generation
Zhang, J., Hu, X., Gao, S., Xia, X., Lo, D., & Li, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715778 · Cited by 19
“The validation dataset, Defects4J [34], evaluates the performance of the fine-tuned LLMs.”
Fault Localization from the Semantic Code Search Perspective
Qin, Y., Wang, S., Lei, Y., Zhang, Z., Lin, B., Peng, X., Ma, J., Chen, L., & Mao, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3757915 · Cited by 15
“The knowledge about the Defects4J [22] and the GrowingBugs [19] benchmark may have been included in the LLM training corpus.”
InfluentialA hybrid evolutionary system for automatic software repair
Yuan, Y., & Banzhaf, W. (2019) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3321707.3321830 · Cited by 19
Checked coverage for test suite reduction: is it worth the effort?
Koitz, R., Stracke, L., & Wotawa, F. (2022) · doi:10.1145/3524481.3527216 · Cited by 14
Metamorphic Testing with Causal Graphs
Clark, A. G., Foster, M. S., Walkinshaw, N., & Hierons, R. M. (2023) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst57152.2023.00023 · Cited by 19
“As a basis for our preliminary analysis, we refer to the analysis carried out by Sobreira et al. [18], who classified the fixes for each bug in the Defects4J repository [19] (which contains details of 395 bugs from six open source Java projects).”
MethodHenderson, T. A. D., Dorward, B., Nickell, E., Johnston, C., & Kondareddy, A. (2023) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst57152.2023.00041 · Cited by 14
“Defects4J [24]), bug databases [25], [26], [27], and synthetic benchmarks with injected bugs [28], [29].”
BackgroundIdentifying Coincidental Correct Test Cases with Multiple Features Extraction for Fault Localization
Wu, Y., Tian, S., Yang, Z., Li, Z., Liu, Y., & Chen, X. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00109 · Cited by 10
“Defects4J 1 , widely used in fault localization studies, contains single-fault and multi-fault programs with real errors and the corresponding CC test cases [28].”
BackgroundPractical Program Repair via Preference-based Ensemble Strategy
Zhong, W., Li, C., Liu, K., Xu, T., Ge, J., Bissyandé, T. F., Luo, B., & Ng, V. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623310 · Cited by 20
“2 [16] for the data collection in previous section, and Bears [27] for an empirical experiment in RQ3.”
BackgroundExpressAPR: Efficient Patch Validation for Java Automated Program Repair Systems
Xiao, Y.-A., Yang, C., Wang, B., & Xiong, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00012 · Cited by 15
“ExpressAPR provides out-of-the-box support for the Defects4J [14] benchmark and the Maven build tool.”
BackgroundMethodTowards LLM-Generated Code Tours for Onboarding
Balfroid, M., Vanderose, B., & Devroey, X. (2024) · doi:10.1145/3643787.3648033 · Cited by 14
“For this case study, we use the well-known Defects4j dataset [9], which contains a diverse and extensive range of actual bugs.”
MethodAutomated search for good coverage criteria
McMinn, P., Harman, M., Fraser, G., & Kapfhammer, G. M. (2016) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1145/2897010.2897013 · Cited by 10
Evaluating Test Suite Effectiveness and Assessing Student Code via Constraint Logic Programming
Dewey, K. T., Conrad, P. T., Craig, M., & Morozova, E. (2017) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3059009.3059051 · Cited by 10
Effective online software anomaly detection
Chen, Y., Ying, M., Liu, D., Alim, A., Feng, C., & Chen, M.-H. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092730 · Cited by 11
Evaluating the Strategies of Statement Selection in Automated Program Repair
Yang, D., Qi, Y., & Mao, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-04272-1_3 · Cited by 13
Learning the Relation Between Code Features and Code Transforms With Structured Prediction
Yu, Z., Martínez, M., Chen, Z., Bissyandé, T. F., & Monperrus, M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3275380 · Cited by 17
Almulla, H. K., & Gay, G. (2020) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst46399.2020.00017 · Cited by 10
Ensemble Random Forests Classifier for Detecting Coincidentally Correct Test Cases
Dass, S., Xue, X., & Namin, A. S. (2020) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac48688.2020.00-72 · Cited by 10
A comprehensive study of code-removal patches in automated program repair
Ginelli, D., Martínez, M., Mariani, L., & Monperrus, M. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10100-7 · Cited by 13
Duets: A Dataset of Reproducible Pairs of Java Library-Clients
Durieux, T., Soto-Valero, C., & Baudry, B. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00071 · Cited by 10
Evaluating code‐based test input generator tools
Cseppentő, L., & Micskei, Z. (2017) · Software Testing Verification and Reliability · doi:10.1002/stvr.1627 · Cited by 11
On the use of sequence mining within spectrum based fault localisation
Laghari, G., & Demeyer, S. (2018) · ACM Symposium on Applied Computing · doi:10.1145/3167132.3167337 · Cited by 10
Cordy, M., Rwemalika, R., Papadakis, M., & Harman, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.03197 · Cited by 12
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 50
A Systematic Exploration of Mutation‐Based Fault Localization Formulae
Wang, B., Wei, J., Chen, M., Chen, C., Lin, Y., & Zhang, J. M. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1905 · Cited by 11
“To alleviate this threat, we directly use the most popular benchmark for Java (i.e., Defects4J [32]) in the field of FL.”
InfluentialA Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing Techniques
Elgendy, I. T., Hierons, R. M., & McMinn, P. (2025) · Software Testing Verification and Reliability · doi:10.1002/stvr.1914 · Cited by 12
“Notable datasets and frameworks used include Defects4J [216] (Java), Software-artefact Infrastructure Repository [221] (C), TravisTorrent [222] (CI platforms), NL- RX-Synth [223] (regular expressions for natural language) and BugsInPy [224] (Python).”
AdverIntent-Agent: Adversarial Reasoning for Repair Based on Inferred Program Intent
Ye, H., Yang, A. Z. H., Hu, C., Wang, Y., Zhang, T., & Goues, C. L. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728939 · Cited by 31
“Weimplement AdverIntent-Agent and evaluate it against two benchmarks: Defects4J 2.0 [22] and HumanEval-Java [20].”
InfluentialClarifying Semantics of In-Context Examples for Unit Test Generation
Yang, C., Yang, L., Wang, Z., Wang, D., Zhou, J., & Chen, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00250 · Cited by 14
“Following the existing work [20], we used the latest versions of four Java projects in the widely-used Defects4J benchmark [21] (i.e., Chart, Time, Lang, and Math), excluding Closure due to its lack of JUnit tests and high testing costs.”
A Comprehensive Study on Large Language Models for Mutation Testing
Wang, B., Chen, M., Deng, M., Lin, Y., Harman, M., Papadakis, M., & Zhang, J. M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3805038 · Cited by 35
“To this end, we employ the Defects4J 2.0 [40] and ConDefects [89] to evaluate the mutation generation approaches, shown as Table 3.”
InfluentialMutant Reduction Evaluation: What is There and What is Missing?
Zhang, P., Wang, Y., Liu, X., Li, Y., Yang, Y., Wang, Z., Zhou, X., Chen, L., & Zhou, Y. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3522578 · Cited by 9
A comprehensive empirical investigation on failure clustering in parallel debugging
Song, Y., Xie, X., Liu, Q., Zhang, X., & Wu, X. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111452 · Cited by 10
GBGallery : A benchmark and framework for game testing
Li, Z., Wu, Y., Ma, L., Xie, X., Chen, Y., & Fan, C. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10158-x · Cited by 11
Trautsch, A., Herbold, S., & Grabowski, J. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10301-2 · Cited by 9
Jiang, S. Q., Zhang, J., Chen, W., Wang, B., Zhou, J., & Zhang, J. M. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648390 · Cited by 16
“More specifically, we performed a preliminary case study on 120 real-world Java bugs from six projects of the benchmark Defects4J [6], by comparing three commercial closed-source LLMs, namely ChatGPT 3.5, ERNIE Bot 3.5, and IFlytek Spark 2.0.”
InfluentialSaha, R. K., Yoshida, H., Prasad, M. R., Tokumoto, S., Takayama, K., & Nanba, I. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3183487 · Cited by 9
Robustness of spectrum-based fault localisation in environments with labelling perturbations
Xu, Y., Yin, B., Zheng, Z., Zhang, X., Li, C., & Yang, S. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.09.091 · Cited by 9
Program Repair at Arbitrary Fault Depth
Khaireddine, B., Martínez, M., & Mili, A. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00056 · Cited by 10
Hao, G., Li, F., Huo, W., Sun, Q., Wang, W., Li, X., & Zou, W. (2019) · International Symposium on Theoretical Aspects of Software Engineering (TASE) · doi:10.1109/tase.2019.00-18 · Cited by 9
An Industrial Study on the Differences between Pre-Release and Post-Release Bugs
Rwemalika, R., Kintis, M., Papadakis, M., Traon, Y. L., & Lorrach, P. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00019 · Cited by 9
Precfix: large-scale patch recommendation by mining defect-patch pairs
Zhang, X., Zhu, C., Li, Y., Guo, J., Liu, L., & Gu, H. (2020) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3377813.3381356 · Cited by 13
Pan, Y., Xiao, X., Hu, G., Zhang, B., Li, Q., & Zheng, H.-T. (2020) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom50675.2020.00107 · Cited by 9
Ahead of Time Mutation Based Fault Localisation using Statistical Inference
Kim, J.-H., An, G., FELDT, R. H., & Yoo, S. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00036 · Cited by 11
Method Calls Frequency-Based Tie-Breaking Strategy For Software Fault Localization
Sarhan, Q. I., Vancsics, B., & Beszédes, Á. (2021) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam52516.2021.00021 · Cited by 9
Identifying Incorrect Patches in Program Repair Based on Meaning of Source Code
Phung, Q.-N., Kim, M., & Lee, E. (2022) · IEEE Access · doi:10.1109/access.2022.3145983 · Cited by 15
Greca, R., Miranda, B., Gligoric, M., & Bertolino, A. (2022) · doi:10.1145/3524481.3527223 · Cited by 9
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.