Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 201–300 of 1,012
Better Patching Using LLM Prompting, via Self-Consistency
Ahmed, T., & Devanbu, P. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00065 · Cited by 41
“Program repair is a popular topic; conducting a comprehensive comparison of all the models [16]–[20] and datasets [15], [21], [22] is beyond the scope of this paper; we’re interested to see if commit log messages can be used as chain-of-thought for applying self-consistency.”
BackgroundCommon Bug-Fix Patterns: A Large-Scale Observational Study
Campos, E. C., & Maia, M. D. A. (2017) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2017.55 · Cited by 43
Network-Clustered Multi-Modal Bug Localization
Hoang, T., Oentaryo, R. J., Le, T.-D. B., & Lo, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2810892 · Cited by 46
Gu, Y., Xuan, J., Zhang, H., Zhang, L., Fan, Q., Xie, X., & Qian, T. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.11.004 · Cited by 42
HMER: A Hybrid Mutation Execution Reduction approach for Mutation-based Fault Localization
Li, Z., Wang, H., & Liu, Y. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110661 · Cited by 44
SpongeBugs: Automatically generating fix suggestions in response to static code analysis warnings
Marcílio, D., Furia, C. A., Bonifácio, R., & Pinto, G. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110671 · Cited by 46
CREF: An LLM-Based Conversational Software Repair Framework for Programming Tutors
Yang, B., Tian, H., Pian, W., Yu, H., Wang, H., Klein, J., Bissyandé, T. F., & Jin, S. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680328 · Cited by 41
“This process is executed in at most three turns of dialogs, and has been validated on 337 bugs from QuixBugs and Defects4J.”
MethodInfluentialThe Plastic Surgery Hypothesis in the Era of Large Language Models
Xia, C. S., Ding, Y., & Zhang, L. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00047 · Cited by 67
“On the widely studied Defects4j 1.2 and 2.0 datasets [43], FitRepair is able to achieve the new state-of-the-art results of 89 and 44 correct bug fixes (15 and 8 more than best baseline) respectively.”
InfluentialDomain Adaptation for Code Model-Based Unit Test Case Generation
Shin, J., Hashtroudi, S., Hemmati, H., & Wang, S. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680354 · Cited by 40
“(2) We conducted an empirical study on Defects4j benchmark dataset [14], which shows our approach improves the performance of the most related work AthenaTest , A3Test , and GPT-4 ) from the literature.”
InfluentialHow Do Automatically Generated Unit Tests Influence Software Maintenance?
Shamshiri, S., Rojas, J. M., Galeotti, J. P., Walkinshaw, N., & Fraser, G. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00033 · Cited by 61
Extracting Concise Bug-Fixing Patches from Human-Written Patches in Version Control Systems
Jiang, Y., Liu, H., Niu, N., Zhang, L., & Hu, Y. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00069 · Cited by 41
JBSE: a symbolic executor for Java programs with complex heap inputs
Braione, P., Denaro, G., & Pezzè, M. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2983940 · Cited by 39
Coming: A Tool for Mining Change Pattern Instances from Git Commits
Martínez, M., & Monperrus, M. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00043 · Cited by 39
Salahirad, A., Almulla, H. K., & Gay, G. (2019) · Software Testing Verification and Reliability · doi:10.1002/stvr.1701 · Cited by 39
Improving deep‐learning‐based fault localization with resampling
Zhang, Z., Lei, Y., Mao, X., Yan, M. L., Xu, L., & Wen, J. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2312 · Cited by 46
Break-It-Fix-It: Unsupervised Learning for Program Repair
Yasunaga, M., & Liang, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06600 · Cited by 27
µBert: Mutation Testing using Pre-Trained Language Models
Degiovanni, R., & Papadakis, M. (2022) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw55395.2022.00039 · Cited by 50
Accuracy Graphs of Spectrum-Based Fault Localization Formulas
Tang, C. M., Chan, W. K., Yu, Y. T., & Zhang, Z. (2017) · IEEE Transactions on Reliability · doi:10.1109/tr.2017.2688487 · Cited by 39
Jaguar: A Spectrum-Based Fault Localization Tool for Real-World Software
Ribeiro, H. L., de Araujo, R. P. A., Chaim, M. L., de Souza, H. A., & Kon, F. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00048 · Cited by 40
Open Science in Software Engineering: A Study on Deep Learning-Based Vulnerability Detection
Nong, Y., Sharma, R., Hamou‐Lhadj, A., Luo, X., & Cai, H. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3207149 · Cited by 47
“In software defect analysis, researchers usually use standard real-world baselines/datasets like Defect4J [105] and Bugs.”
CodeGen4Libs: A Two-Stage Approach for Library-Oriented Code Generation
Liu, M., Yang, T., Lou, Y., Du, X., Wang, Y., & Peng, X. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00159 · Cited by 71
A Theoretical and Empirical Study of Diversity-Aware Mutation Adequacy Criterion
Shin, D., Yoo, S., & Bae, D. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2732347 · Cited by 37
SketchFix: a tool for automated program repair approach using lazy candidate generation
Hua, J., Zhang, M., Wang, K., & Khurshid, S. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264600 · Cited by 36
PraPR: Practical Program Repair via Bytecode Mutation
Ghanbari, A., & Zhang, L. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00116 · Cited by 43
HyDiff: hybrid differential software analysis
Noller, Y., Păsăreanu, C. S., Böhme, M., Sun, Y., Nguyen, H. L., & Grunske, L. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380363 · Cited by 44
State of Practical Applicability of Regression Testing Research: A Live Systematic Literature Review
Greca, R., Miranda, B., & Bertolino, A. (2023) · ACM Computing Surveys · doi:10.1145/3579851 · Cited by 42
Context-Aware Neural Fault Localization
Zhang, Z., Lei, Y., Mao, X., Yan, M. L., Xia, X., & Lo, D. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3279125 · Cited by 38
“The first 4 programs [61] (i.e., chart , math , lang , and time ) are from Defects4J.”
BackgroundTest Oracle Automation in the Era of LLMs
Molina, F., Gorla, A., & d’Amorim, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715107 · Cited by 73
“To illustrate this issue, let us consider Defects4J [17], one the most widely-adopted benchmarks in software testing research.”
TOGLL: Correct and Strong Test Oracle Generation with LLMS
Hossain, S. B., & Dwyer, M. B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00098 · Cited by 110
“These latter projects are characterized by large codebases with multiple modules, with up to 10k active users per project, and are frequently employed in software engineering research for the evaluation of test cases and test oracles [6], [7], [36].”
InfluentialUncertainty-Driven Black-Box Test Data Generation
Walkinshaw, N., & Fraser, G. (2017) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2017.30 · Cited by 37
Majd, A., Vahidi-Asl, M., Khalilian, A., Baraani-Dastjerdi, A., & Zamani, B. (2019) · Journal of Computer Languages · doi:10.1016/j.cola.2019.03.006 · Cited by 49
Improving Software Fault Localization by Combining Spectrum and Mutation
Cui, Z., Jia, M., Chen, X., Zheng, L., & Liu, X. (2020) · IEEE Access · doi:10.1109/access.2020.3025460 · Cited by 36
A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark
Ye, H., Martínez, M., Durieux, T., & Monperrus, M. (2019) · IEEE International Workshop on Intelligent Bug Fixing (IBF) · doi:10.1109/ibf.2019.8665475 · Cited by 41
Yu, Z. (., Martínez, M., Danglot, B. (., Durieux, T., & Monperrus, M. (2018) · LillOA (Université de Lille (University Of Lille)) · Cited by 36
Generating Effective Test Suites by Combining Coverage Criteria
Gay, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66299-2_5 · Cited by 36
Template-based Neural Program Repair
Meng, X., Wang, X., Zhang, H., Sun, H., Liu, X., & Hu, C. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00127 · Cited by 65
“0 [30] in our experiments, which are widely used in automated program repair studies [5], [18]–[22], [26]”
BackgroundMethodLearning to Generate Corrective Patches using Neural Machine TranslationPreprint
Hata, H., Shihab, E., & Neubig, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07170 · Cited by 63
Wang, H., Du, B., He, J., Liu, Y., & Chen, X. (2020) · IEEE Access · doi:10.1109/access.2020.3004145 · Cited by 35
The Fitness Function for the Job: Search-Based Generation of Test Suites That Detect Real Faults
Gay, G. (2017) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2017.38 · Cited by 35
Revisiting and Improving SZZ Implementations
Neto, E. C., da Costa, D. A., & Kulesza, U. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870178 · Cited by 36
Tian, H., Liu, K., Li, Y., Kaboré, A., Koyuncu, A., Habib, A., Li, L., Wen, J.-H., Klein, J., & Bissyandé, T. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3576039 · Cited by 51
Neural-Based Test Oracle Generation: A Large-Scale Evaluation and Lessons Learned
Hossain, S. B., Filieri, A., Dwyer, M. B., Elbaum, S. G., & Visser, W. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616265 · Cited by 60
“The bug-detection effectiveness of the TOGA-generated test oracles was evaluated on the Defects4j benchmark, a dataset of real bugs [29], consisting of 17 Java applications containing a total of 835 labeled bugs.”
BackgroundMethodInfluentialComparing repositories visually with repograms
Rozenberg, D., Beschastnikh, I., Kosmale, F., Poser, V., Becker, H., Palyart, M., & Murphy, G. C. (2016) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2901739.2901768 · Cited by 34
Three Open Problems in the Context of E2E Web Testing and a Vision: NEONATE
Ricca, F., Leotta, M., & Stocco, A. (2018) · Advances in computers · doi:10.1016/bs.adcom.2018.10.005 · Cited by 34
Wang, S., Wen, M., Chen, L., Yi, X., & Mao, X. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870172 · Cited by 43
On the effectiveness of unified debugging: an extensive study on 16 program repair systems
Benton, S., Li, X., Lou, Y., & Zhang, L. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416566 · Cited by 37
SCELMo: Source Code Embeddings from Language ModelsPreprint
Karampatsis, R.-M., & Sutton, C. (2020) · arXiv.org · doi:10.48550/arxiv.2004.13214 · Cited by 60
Alikhanifard, P., & Tsantalis, N. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3696002 · Cited by 45
Astor: Exploring the design space of generate-and-validate program repair beyond GenProg
Martínez, M., & Monperrus, M. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.01.069 · Cited by 43
Evaluating Automatic Program Repair Capabilities to Repair API Misuses
Kechagia, M., Mechtaev, S., Sarro, F., & Harman, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3067156 · Cited by 43
Predicting Patch Correctness Based on the Similarity of Failing Test Cases
Tian, H., Li, Y., Pian, W., Kaboré, A. K., Liu, K., Habib, A., Klein, J., & Bissyandé, T. F. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3511096 · Cited by 49
A program slicing-based method for effective detection of coincidentally correct test cases
Feyzi, F., & Parsa, S. A. (2018) · Computing · doi:10.1007/s00607-018-0591-z · Cited by 34
Tare: Type-Aware Neural Program Repair
Zhu, Q., Sun, Z., Zhang, W., Xiong, Y., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00126 · Cited by 72
“2 [24], additional bugs from Defects4J v2.”
SoapFL: A Standard Operating Procedure for LLM-Based Method-Level Fault Localization
Qin, Y.-H., Wang, S.-W., Lou, Y., Dong, J., Wang, K.-X., Li, X.-L., & Mao, X.-G. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3543187 · Cited by 56
“2.0 [20] benchmark, note that GNet4FL [23] has not been evaluated on project Closure.”
InfluentialCalibration and Correctness of Language Models for Code
Spieß, C. K., Gros, D., Pai, K. S., Pradel, M., Rabin, M. R. I., Alipour, M. A., Jha, S., Dévanbu, P., & Ahmed, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00040 · Cited by 120
“We leverage the widely-used Defects4J dataset [41], which includes real-world examples of buggy programs, with fixes and test-sets.”
GraphCode2Vec: generic code embedding via lexical and program dependence analyses
Ma, W., Zhao, M., Soremekun, E., Hu, Q., Zhang, J. M., Papadakis, M., Cordy, M., Xie, X., & Traon, Y. L. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528456 · Cited by 54
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31
Path-directed source test case generation and prioritization in metamorphic testing
Sun, C., Liu, B., Fu, A., Liu, Y., & Liu, H. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111091 · Cited by 30
Automated unit test generation for evolving software
Shamshiri, S. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2803196 · Cited by 31
Symbolic execution of programs with heap inputs
Braione, P., Denaro, G., & Pezzè, M. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786842 · Cited by 31
DroidLeaks: a comprehensive database of resource leaks in Android apps
Liu, Y., Wang, J., Wei, L., Xu, C., Cheung, S.-C., Wu, T., Yan, J., & Zhang, J. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09715-8 · Cited by 31
Effective and efficient API misuse detection via exception propagation and search-based testing
Kechagia, M., Devroey, X., Panichella, A., Gousios, G., & Arie Van Deursen, A. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330552 · Cited by 31
Automatic patch generation with context-based change application
Kim, J., & Kim, S. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09742-5 · Cited by 39
A manual inspection of Defects4J bugs and its implications for automatic program repair
Jiang, J., Xiong, Y., & Xia, X. (2019) · Science China Information Sciences · doi:10.1007/s11432-018-1465-6 · Cited by 41
Iterative User-Driven Fault Localization
Li, X., d’Amorim, M., & Orso, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49052-6_6 · Cited by 31
APPT: Boosting Automated Patch Correctness Prediction via Fine-Tuning Pre-Trained Models
Zhang, Q., Fang, C., Sun, W., Liu, Y., He, T., Hao, X., & Chen, Z. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3354969 · Cited by 57
A benchmark-based evaluation of search-based crash reproduction
Soltani, M., Derakhshanfar, P., Devroey, X., & Arie Van Deursen, A. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09762-1 · Cited by 31
Towards More Realistic Evaluation for Neural Test Oracle Generation
Liu, Z., Liu, K., Xia, X., & Yang, X. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598080 · Cited by 40
“Defects4J [22] is a widely-used bug benchmark containing 835 bugs from 17 real-world Java projects.”
BackgroundInfluentialDoes data sampling improve deep learning-based vulnerability detection? Yeas! and Nays!
Yang, X., Wang, S., Li, Y., & Wang, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00192 · Cited by 56
“Prior studies report that there is a signifi-cant portion of similar code in code corpus [45]–[47].”
BackgroundBehavioral Execution Comparison: Are Tests Representative of Field Behavior?
Wang, Q., Brun, Y., & Orso, A. (2017) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2017.36 · Cited by 29
SecBench.js: An Executable Security Benchmark Suite for Server-Side JavaScript
Bhuiyan, M. H. M., Parthasarathy, A. S., Vasilakis, N., Pradel, M., & Staicu, C.-A. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00096 · Cited by 35
“For example, such benchmarks serve to assess the relative merits of different contributions in databases [27], parallel processing [28], graphics [29], machine learning [30], and bug detection and repair [31].”
BackgroundInfluentialZhao, P., Miao, Z., Lan, S., & Zhao, J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111805 · Cited by 56
Analyzing Quantum Programs with LintQ: A Static Analysis Framework for Qiskit
Paltenghi, M., & Pradel, M. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660802 · Cited by 61
“While this number may seem low, it is actually higher than the recall of popular static bug detectors on Defect4J [29], which prior work has measured to be between 1% and 3%, depending on the bug detector [25].”
Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction
Kang, S., Yoon, J., Askarbekkyzy, N., & Yoo, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3450837 · Cited by 54
“Our experiments reveal that when using the code-davinci-002 model from OpenAI, L IBRO can reproduce one-third of all the bugs in the widely used Defects4J benchmark [9], substantially outperforming bug reproduction baselines that we compared against.”
MethodFight Fire With Fire: How Much Can We Trust ChatGPT on Source Code-Related Tasks?
Yu, X., Liu, L., Hu, X., Keung, J., Liu, J., & Xia, X. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3492204 · Cited by 61
“We acknowledge the widespread use of two classic program repair datasets, Defects4J [32] and ManyBugs [33] in program repair research.”
Oracle problem in software testing
Jahangirova, G. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3098235 · Cited by 34
Using a probabilistic model to predict bug fixes
Soto, M., & Goues, C. L. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330211 · Cited by 30
Break the dead end of dynamic slicing: localizing data and control omission bug
Lin, Y., Sun, J., Tran, L., Bai, G., Wang, H., & Dong, J. S. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238163 · Cited by 28
Explaining Regressions via Alignment Slicing and Mending
Wang, H., Lin, Y., Yang, Z., Sun, J., Liu, Y., Dong, J. S., Zheng, Q., & Liu, T. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2949568 · Cited by 29
Fast and Precise On-the-Fly Patch Validation for All
Chen, L., Ouyang, Y., & Zhang, L. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00104 · Cited by 32
Enhancing Spectrum-Based Fault Localization Using Fault Influence Propagation
He, H., Ren, J., Zhao, G., & He, H. (2020) · IEEE Access · doi:10.1109/access.2020.2965139 · Cited by 29
The Remarkable Role of Similarity in Redundancy-based Program RepairPreprint
Chen, Z., & Monperrus, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.05703 · Cited by 39
Fight Fire with Fire: How Much Can We Trust ChatGPT on Source Code-Related Tasks?
Yu, X., Liu, L., Hu, X., Keung, J. W., Liu, J., & Xia, X. (2024) · arXiv · doi:10.48550/arxiv.2405.12641 · Cited by 29
Comprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Zhang, J., Bao, X., Wang, X., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3532759 · Cited by 56
“For example, CodeT5 and UniXcoder achieved the best repair results using raw code input, while CodeBERT and GraphCodeBERT have close results on abs and raw .”
Learning how to search: generating effective test cases through adaptive fitness function selection
Almulla, H. K., & Gay, G. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10048-8 · Cited by 41
Yang, D., Liu, K., Kim, D., Koyuncu, A., Kim, K., Tian, H., Lei, Y., Mao, X., Klein, J., & Bissyandé, T. F. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10003-7 · Cited by 39
Studying Test Case Failure Prediction for Test Case Prioritization
Noor, T. B., & Hemmati, H. (2017) · doi:10.1145/3127005.3127006 · Cited by 28
Just, R., Parnin, C., Drosos, I., & Ernst, M. D. (2018) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3213846.3213870 · Cited by 29
Self-citationCoincidental correctness in the Defects4J benchmark
Assi, R. A., Trad, C., Maalouf, M., & Masri, W. (2019) · Software Testing Verification and Reliability · doi:10.1002/stvr.1696 · Cited by 28
Exploring the Differences between Plausible and Correct Patches at Fine-Grained Level
Yang, B. K., & Yang, J. (2020) · IEEE International Workshop on Intelligent Bug Fixing (IBF) · doi:10.1109/ibf50092.2020.9034821 · Cited by 39
Java Unit Testing Tool Competition - Fifth Round
Panichella, A., & Rueda, U. (2017) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1109/sbst.2017.7 · Cited by 28
Wu, Y., Li, Z., Zhang, J. M., & Liu, Y. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663815 · Cited by 51
“ConDefects includes Python and Java programs to align with established datasets in the field [9, 10, 23].”
A Characterization Study of Repeated Bug Fixes
Yue, R., Meng, N., & Wang, Q. (2017) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2017.16 · Cited by 28
Pradhan, D., Wang, S., Ali, S., Yue, T., & Liaaen, M. C. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2882176 · Cited by 27
Empirical Evaluation of Fault Localisation Using Code and Change Metrics
Sohn, J., & Yoo, S. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2930977 · Cited by 33
ABFL: An autoencoder based practical approach for software fault localization
Peng, Z., Xiao, X., Hu, G., Sangaiah, A. K., Atiquzzaman, M., & Xia, S. (2019) · Information Sciences · doi:10.1016/j.ins.2019.08.077 · Cited by 34
CirFix: automatically repairing defects in hardware design code
Ahmad, H., Huang, Y., & Weimer, W. R. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3503222.3507763 · Cited by 52
Exploring Better Black-Box Test Case Prioritization via Log Analysis
Chen, Z., Chen, J., Wang, W., Zhou, J., Wang, M., Chen, X., Zhou, S., & Wang, J. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3569932 · Cited by 43
Artificial Intelligence for Software Engineering: The Journey So Far and the Road Ahead
Ahmed, I., Aleti, A., Cai, H., Chatzigeorgiou, A., He, P., Hu, X., Pezzè, M., Poshyvanyk, D., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719006 · Cited by 51
Ant Colony System With Sorting-Based Local Search for Coverage-Based Test Case Prioritization
Lu, C., Zhong, J., Xue, Y., Feng, L., & Zhang, J. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2019.2930358 · Cited by 26
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.