Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 401–500 of 1,012
FPA-FL: Incorporating static fault-proneness analysis into statistical fault localization
Feyzi, F., & Parsa, S. A. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.11.002 · Cited by 18
Are mutants really natural? a study on how "naturalness" helps mutant selection
Jimenez, M., Checkam, T. T., Cordy, M., Papadakis, M., Kintis, M., Traon, Y. L., & Harman, M. (2018) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3239235.3240500 · Cited by 25
Can We Predict the Quality of Spectrum-based Fault Localization?
Golagha, M., Pretschner, A., & Briand, L. (2020) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst46399.2020.00012 · Cited by 19
Automatic repair of OWASP Top 10 security vulnerabilities
Marchand-Melsom, A., & Nguyen, D. B. (2020) · doi:10.1145/3387940.3392200 · Cited by 18
Test Case Generation for Program Repair: A Study of Feasibility and EffectivenessPreprint
Yu, Z., Martínez, M., Danglot, B., Durieux, T., & Monperrus, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00198 · Cited by 18
COCA: Generative Root Cause Analysis for Distributed Systems with Code Knowledge
Li, Y., Wu, Y., Liu, J., Jiang, Z., Chen, Z., Yu, G., & Lyu, M. R.-T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00234 · Cited by 49
“However, bug reproduction-based fault localization techniques [10]–[12] face challenges for localizing the faulty code in distributed systems, which is extremely challenging with only issue reports available [9], [13].”
A Light-Weight Data Augmentation Method for Fault Localization
Hu, J., Xie, H., Lei, Y., & Yu, K. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2023.107148 · Cited by 18
REEF: A Framework for Collecting Real-World Vulnerabilities and Fixes
Wang, C., Li, Z., Pena, Y., Gao, S., Chen, S., Wang, S., Gao, C., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00199 · Cited by 33
“TypeFix [19] is a prompt-based approach with fix templates incorporated for repairing Python type errors.”
MethodProgram Repair Guided by Datalog-Defined Static Analysis
Liu, Y., Mechtaev, S., Subotić, P., & Roychoudhury, A. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616363 · Cited by 30
“We constructed datasets specifically for the defect classes handled by the Datalog analysers, instead of relying on Defects4J [20] and QuixBugs [26] often used for evaluating test-driven repair.”
MethodStatfier: Automated Testing of Static Analyzers via Semantic-Preserving Program Transformations
Zhang, H., Pei, Y., Chen, J., & Tan, S. H. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616272 · Cited by 17
“Another alternative is to adopt Defects4J [43], which is a widely used dataset with defects for Java programs.”
BackgroundWang, S., Wen, M., Mao, X., & Yang, D. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319349 · Cited by 18
A Dataset of Non-Functional Bugs
Radu, A., & Nadi, S. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00066 · Cited by 22
Fault localization based on wide & deep learning model by mining software behavior
Wang, T., Yu, H., Wang, K., & Su, X. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.09.026 · Cited by 18
Efficiently finding higher-order mutants
Wong, C.-P., Meinicke, J., Chen, L., Diniz, J. P., Kästner, C., & Figueiredo, E. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409713 · Cited by 17
Feyzi, F. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09859-y · Cited by 17
Towards an automated approach for bug fix pattern detectionPreprint
Madeiral, F., Durieux, T., Sobreira, V., & Maia, M. D. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11286 · Cited by 17
A3test: Assertion-Augmented Automated Test Case GenerationPreprint
Alagarsamy, S., Tantithamthavorn, C. K., & Aleti, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4724885 · Cited by 21
Aligning the Objective of LLM-Based Program Repair
Xu, J., Fu, Y., Tan, S. H., & He, P. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00169 · Cited by 69
“Defects4J [36] is a widely used APR benchmark with 835 real-world bugs from 17 open-source repositories.”
InfluentialUsing Relative Lines of Code to Guide Automated Test Generation for Python
Holmes, J., Ahmed, I., Brindescu, C., Gopinath, R., Zhang, H., & Groce, A. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3408896 · Cited by 20
A Comprehensive Study on Quality Assurance Tools for Java
Liu, H., Chen, S., Feng, R., Liu, C., Li, K., Xu, Z., Nie, L., Liu, Y., & Chen, Y. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598056 · Cited by 28
“For example, regarding project Jsoup with bug No.4 of benchmark Defects4J, Infer reported a warning that m.group(1) may be null, which may not be the reason for this bug.”
InfluentialEvaluating Search-Based Software Microbenchmark Prioritization
Laaber, C., Yue, T., & Ali, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3380836 · Cited by 16
Evolving Paradigms in Automated Program Repair: Taxonomy, Challenges, and Opportunities
Huang, K., Xu, Z., Yang, S., Sun, H., Li, X., Yan, Z., & Zhang, Y. (2024) · ACM Computing Surveys · doi:10.1145/3696450 · Cited by 111
“Yang et al. [120] found that even on the widely used Defects4J [35], the patch data is not optimized enough.”
Are Large Language Models Memorizing Bug Benchmarks?
Ramos, D., Mamede, C., Jain, K., Canelas, P., Gamboa, C., & Goues, C. L. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00005 · Cited by 52
“Notable such datasets include, for example, Defects4J [4] ( Java ) and BugsInPy [5] ( Python ); similarly, ML researchers recently introduced SWEBench [6].”
Candoia: A Platform for Building and Sharing Mining Software Repositories Tools as Apps
Tiwari, N. M., Upadhyaya, G., Nguyen, H. A., & Rajan, H. (2017) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.56 · Cited by 16
Restore: Retrospective Fault Localization Enhancing Automated Program Repair
Xu, T., Chen, L., Pei, Y., Zhang, T. Y., Pan, M., & Furia, C. A. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2987862 · Cited by 20
CounterFault: Value-Based Fault Localization by Modeling and Predicting Counterfactual Outcomes
Podgurski, A., & Kucuk, Y. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00044 · Cited by 23
Ajibode, A., Shu, T., & Ding, Z. (2020) · IEEE Access · doi:10.1109/access.2020.3035413 · Cited by 16
Combining Query Reduction and Expansion for Text-Retrieval-Based Bug Localization
Florez, J. M., Chaparro, O., Treude, C., & Marcus, A. (2021) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner50967.2021.00024 · Cited by 16
More Reliable Test Suites for Dynamic APR by using Counterexamples
Nilizadeh, A., Calvo, M., Leavens, G. T., & Le, X.-B. D. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00032 · Cited by 20
Styler: learning formatting conventions to repair Checkstyle violations
Loriot, B., Madeiral, F., & Monperrus, M. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10107-0 · Cited by 31
An empirical study of inadequate and adequate test suite reduction approaches
Coviello, C., Romano, S., & Scanniello, G. (2018) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3239235.3240497 · Cited by 16
FTMES: A Failed-Test-Oriented Mutant Execution Strategy for Mutation-Based Fault Localization
de Oliveira, A. A. L., Camilo-Junior, C. G., Freitas, E. N. D. A., & Vincenzi, A. M. R. (2018) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2018.00026 · Cited by 17
Neural Bug Finding: A Study of Opportunities and ChallengesPreprint
Habib, A., & Pradel, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00307 · Cited by 30
DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code SkeletonsPreprint
Drain, D., Clement, C. B., Serrato, G., & Sundaresan, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09352 · Cited by 55
Are Large Language Models Memorizing Bug Benchmarks?
Ramos, D., Mamede, C., Jain, K., Canelas, P., Gamboa, C., & Goues, C. L. (2024) · arXiv · doi:10.48550/arxiv.2411.13323 · Cited by 16
When Deep Learning Meets Information Retrieval-based Bug Localization: A Survey
Niu, F., Li, C., Liu, K., Xia, X., & Lo, D. (2025) · ACM Computing Surveys · doi:10.1145/3734217 · Cited by 28
“We note that mainstream research continues to concentrate on the Java language, while only a limited number explore other languages [23, 123].”
Doc2OracLL: Investigating the Impact of Documentation on LLM-Based Test Oracle Generation
Hossain, S. B., Taylor, R., & Dwyer, M. B. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729354 · Cited by 41
“To investigate the generalizability, we additionally perform a study on the unseen Defects4J dataset [23].”
NPEX: repairing Java null pointer exceptions without tests
Lee, J., Hong, S., & Oh, H. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510186 · Cited by 29
Tian, H., Tang, X., Habib, A., Wang, S., Liu, K., Xia, X., Klein, J., & Bissyandé, T. F. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556914 · Cited by 50
How Useful is Code Change Information for Fault Localization in Continuous Integration?
Chen, A. R., Chen, T.-H. (., & Chen, J. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3556931 · Cited by 15
Fonte: Finding Bug Inducing Commits from Failures
An, G., Hong, J., Kim, N., & Yoo, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00059 · Cited by 25
GMBFL: Optimizing Mutation-Based Fault Localization via Graph Representation
Wu, S., Li, Z., Liu, Y., Chen, X., & Li, M. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00033 · Cited by 21
Accelerating Patch Validation for Program Repair With Interception-Based Execution Scheduling
Xiao, Y.-A., Yang, C., Wang, B., & Xiong, Y. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3359969 · Cited by 19
“We used Defects4J [69] v1.”
An empirical study on bug severity estimation using source code metrics and static analysis
Mashhadi, E., Chowdhury, S., Modaberi, S., Hemmati, H., & Uddin, G. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112179 · Cited by 27
Towards efficient and effective automatic program repair
Le, X.-B. D. (2016) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2970276.2975934 · Cited by 16
An Empirical Study on the Usage of Fault Localization in Automated Program Repair
Yang, D., Qi, Y., & Mao, X. (2017) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2017.37 · Cited by 19
Incremental Control Dependency Frontier Exploration for Many-Criteria Test Case Generation
Panichella, A., Kifetew, F. M., & Tonella, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_17 · Cited by 15
Asad, M., Ganguly, K. K., & Sakib, K. M. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00050 · Cited by 21
Akimova, E. N., Bersenev, A. Y., Deikov, A. A., Kobylkin, K. S., Konygin, A. V., Mezentsev, I. P., & Misilov, V. E. (2021) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec53868.2021.00022 · Cited by 18
Discovering common bug‐fix patterns: A large‐scale observational study
Campos, E. C., & Maia, M. D. A. (2019) · Journal of Software Evolution and Process · doi:10.1002/smr.2173 · Cited by 19
Xiaobo, Y., Liu, B., & Wang, S. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110861 · Cited by 15
Leveraging Qualitative Reasoning to Improve SFL
Perez, A., & Abreu, R. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/267 · Cited by 14
Learning Realistic Mutations: Bug Creation for Neural Bug Detectors
Richter, C., & Wehrheim, H. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00027 · Cited by 32
Exploring Parameter-Efficient Fine-Tuning of Large Language Model on Automated Program Repair
Li, G., Zhi, C., Chen, J., Han, J., & Deng, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695066 · Cited by 46
“0 [24], QuixBugs [31], and HumanEval-Java [20] to showcase the effectiveness and efficiency of PEFT in the single-hunk APR task.”
LLMs and Prompting for Unit Test Generation: A Large-Scale Evaluation
Ouédraogo, W. C., Kaboré, A. K., Tian, H., Song, Y., Koyuncu, A., Klein, J., Lo, D., & Bissyandé, T. F. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695330 · Cited by 33
“The evaluation utilized three datasets: SF110 [6], a benchmark with 110 Java projects; Defects4J [10], containing real-world bugs from open-source projects; and a Custom Dataset with Java classes and bugs from recent projects.”
On Inter-Dataset Code Duplication and Data Leakage in Large Language Models
López, J. A. H., Chen, B., Saad, M., Sharma, T., & Varró, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3504286 · Cited by 42
“Through interactions with C HAT GPT, the authors show that C HAT GPT is aware of multiple samples from the D EFECTS 4J dataset [47].”
BackgroundA Comprehensive Survey of Benchmarks for Improvement of Software's Non-Functional Properties
Blot, A., & Petke, J. (2025) · ACM Computing Surveys · doi:10.1145/3711119 · Cited by 23
Advancements in automated program repair: a comprehensive review
Dikici, S., & Bilgin, T. T. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02383-9 · Cited by 39
“Table 2 presents the distribution of bug types across three major benchmarks: Defects4J [51], QuixBugs [52], and BugsJS [53].”
Liu, K., Wang, S., Koyuncu, A., Kim, K., Bissyandé, T. F., Kim, D., Wu, P., Klein, J., Mao, X., & Traon, Y. L. (2020) · Open Repository and Bibliography (University of Luxembourg) · doi:10.48550/arxiv.2008.00914 · Cited by 15
Vancsics, B., Szatmári, A., & Beszédes, Á. (2020) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054803 · Cited by 15
Evaluating the usage of fault localization in automated program repair: an empirical study
Yang, D., Qi, Y., Mao, X., & Lei, Y. (2020) · Frontiers of Computer Science · doi:10.1007/s11704-020-9263-1 · Cited by 18
Enhanced Bug Prediction in JavaScript Programs with Hybrid Call-Graph Based Invocation Metrics
Antal, G., Tóth, Z., Hegedűs, P., & Ferenć, R. (2020) · Technologies · doi:10.3390/technologies9010003 · Cited by 14
A Framework for Improving Fault Localization Effectiveness Based on Fuzzy Expert System
Lin, C.-T., Chen, W., & Intasara, J. (2021) · IEEE Access · doi:10.1109/access.2021.3086878 · Cited by 14
JUGE : An infrastructure for benchmarking Java unit test generators
Devroey, X., Gambi, A., Galeotti, J. P., Just, R., Kifetew, F. M., Panichella, A., & Panichella, S. (2022) · Software Testing Verification and Reliability · doi:10.1002/stvr.1838 · Cited by 15
Self-citationFault centrality: boosting spectrum-based fault localization via local influence calculation
Zhao, G., He, H., & Huang, Y. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02822-4 · Cited by 17
A Practical Approach for Dynamic Taint Tracking with Control-flow Relationships
Hough, K., & Bell, J. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485464 · Cited by 14
Evaluation of fault localization techniques
Pearson, S. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2983967 · Cited by 15
Mining repair model for exception-related bug
Zhong, H., & Mei, H. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.03.046 · Cited by 14
An Analysis on the Negative Effect of Multiple-Faults for Spectrum-Based Fault Localization
Xiaobo, Y., Liu, B., & Wang, S. (2018) · IEEE Access · doi:10.1109/access.2018.2886413 · Cited by 14
Diagnosing Software Faults Using Multiverse Analysis
Chatterjee, P., Chatterjee, A., Campos, J., Abreu, R., & Roy, S. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/226 · Cited by 18
Searching for Multi-fault Programs in Defects4J
An, G., Yoon, J., & Yoo, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88106-1_11 · Cited by 14
AGA: An Accelerated Greedy Additional Algorithm for Test Case Prioritization
Li, F., Zhou, J., Li, Y., Hao, D., & Zhang, L. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3137929 · Cited by 16
A 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13
AugmenTest: Enhancing Tests with LLM-Driven Oracles
Khandaker, S. M., Kifetew, F. M., Prandi, D., & Susi, A. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988926 · Cited by 33
“While TOGA evaluates on datasets such as Defects4J [19] and Methods2Test [20], which are datasets most probably seen by most recent LLMs, potential data leakage is very likely (a concern with TOGA’s reliance on previously trained datasets).”
Automated program repair: a step towards software automation
Roychoudhury, A., & Xiong, Y. (2019) · Science China Information Sciences · doi:10.1007/s11432-019-9947-6 · Cited by 13
Reliable Fix Patterns Inferred from Static Checkers for Automated Program Repair
Liu, K., Zhang, J., Li, L., Koyuncu, A., Kim, D., Ge, C., Liu, Z., Klein, J., & Bissyandé, T. F. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3579637 · Cited by 19
A comprehensive evaluation of SZZ Variants through a developer-informed oracle
Rosa, G., Pascarella, L., Scalabrino, S., Tufano, R., Bavota, G., Lanza, M., & Oliveto, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111729 · Cited by 29
An effective fault localization approach based on PageRank and mutation analysis
Yan, Y. P., Jiang, S., Zhang, Y., & Zhang, C. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111799 · Cited by 14
Automatic Repair of Quantum Programs via Unitary Operation
Li, Y., Pei, H., Huang, L., Yin, B., & Cai, K.-Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3664604 · Cited by 16
“With the development of benchmarks for APR (such as Defects4J [29] for Java programs), several studies like SimFix [27], TBar [32], and Recorder [64] have utilized the previous data or patterns to assist in repairing the current programs.”
Reality Check: Assessing GPT-4 in Fixing Real-World Software Vulnerabilities
Ságodi, Z., Antal, G., Bogenfürst, B., Isztin, M., Hegedűs, P., & Ferenć, R. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661207 · Cited by 23
Iterative Refactoring of Real-World Open-Source Programs with Large Language Models
Choi, J., An, G., & Yoo, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64573-0_4 · Cited by 18
Tests4Py: A Benchmark for System Testing
Smytzek, M., Eberlein, M., Serçe, B., Grunske, L., & Zeller, A. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663798 · Cited by 13
“Notable examples include Defects4J [9], BugsJS [6], and BugsInPy [22] for Java, JavaScript, and Python, respectively.”
BackgroundCoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios
Zeng, Z., Wang, Y., Xie, R., Ye, W., & Zhang, S. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3652115 · Cited by 32
“Automated Program Repair (APR): As previously noted in Section 2.2, the Defects4j [21] dataset is a prevalent source for automated program repair (APR).”
InfluentialDivide-and-Conquer: Automating Code Revisions via Localization-and-Revision
Wang, S., Lin, B., Chen, L., & Mao, X. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697013 · Cited by 15
“For instance, as revealed by the existing study [48], over half of the patches in the Defects4J benchmark [20], which contains hundreds of bug-fixing patches mined from projects on GitHub, involve modifications in more than four statements.”
Deep Learning for Hyperspectral Image Classification: A Critical Evaluation via Mutation Testing
Chen, Z., Hao, Y., Liu, Q., Liu, Y., Zhu, M., & Xiao, L. (2024) · Remote Sensing · doi:10.3390/rs16244695 · Cited by 20
What You See is What You Get: Attention-Based Self-Guided Automatic Unit Test Generation
Yin, X., Ni, C., Xu, X., & Yang, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00105 · Cited by 29
Experiments with Interactive Fault Localization Using Simulated and Real Users
Horváth, F., Beszédes, Á., Vancsics, B., Balogh, G., Vidács, L., & Gyimóthy, T. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00036 · Cited by 13
Denchmark: A Bug Benchmark of Deep Learning-related Software
Kim, M., Kim, Y., & Lee, E. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00070 · Cited by 16
JavaDL: automatically incrementalizing Java bug pattern detection
Dura, A., Reichenbach, C., & Söderberg, E. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485542 · Cited by 13
RegMiner: towards constructing a large regression dataset from code evolution history
Song, X., Lin, Y., Ng, S. H., Wu, Y., Peng, X., Dong, J. S., & Mei, H. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534224 · Cited by 21
Do automated program repair techniques repair hard and important bugs?
Motwani, M., Sankaranarayanan, S., Just, R., & Brun, Y. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182533 · Cited by 17
Self-citationSECBENCH: A Database of Real Security Vulnerabilities.
Reis, S., & Abreu, R. (2017) · Cited by 13
A Theoretical and Empirical Analysis of Program Spectra Diagnosability
Perez, A., Abreu, R., & Arie Van Deursen, A. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2895640 · Cited by 14
Automatically Generating Fix Suggestions in Response to Static Code Analysis Warnings
Marcílio, D., Furia, C. A., Bonifácio, R., & Pinto, G. (2019) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2019.00013 · Cited by 13
Assi, R. A., Masri, W., & Trad, C. (2021) · Software Testing Verification and Reliability · doi:10.1002/stvr.1762 · Cited by 13
ENCORE: Ensemble Learning using Convolution Neural Machine Translation for Automatic Program RepairPreprint
Lutellier, T., Pang, L., Pham, V. H., Wei, M., & Tan, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.08691 · Cited by 22
Zhang, J., Hu, X., Xia, X., Cheung, S.-C., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3745765 · Cited by 20
“The focal method is from the Lang project [11] in the Defects4J benchmark [27], aiming to convert timestamps into string.”
FUM - A Framework for API Usage constraint and Misuse Classification
Schlichtig, M., Sassalla, S., Narasimhan, K., & Bodden, E. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00085 · Cited by 14
Basic block coverage for search-based unit testing and crash reproduction
Derakhshanfar, P., Devroey, X., & Zaidman, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10155-0 · Cited by 14
On the Reproducibility of Software Defect Datasets
Zhu, H., & Rubio-González, C. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00195 · Cited by 25
“First, we conduct a study on reproducibility of five state-of-the-art Java software defect datasets: D EFECTS 4J [30], G ROWING B UGS [29], B UGS .”
MethodInfluential
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.