Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 601–700 of 1,012
Tracking Code Bug Fix Ripple Effects Based on Change Patterns Using Markov Chain Models
Ufuktepe, E., Tuğlular, T., & Palaniappan, K. (2022) · IEEE Transactions on Reliability · doi:10.1109/tr.2022.3167943 · Cited by 9
Deep Learning Based Identification of Suspicious Return Statements
Li, G., Liu, H., Jin, J., & Umer, Q. (2020) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner48275.2020.9054826 · Cited by 10
IRBFL: An Information Retrieval Based Fault Localization Approach
Li, Z., Bai, X. B., Wang, H., & Liu, Y. (2020) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac48688.2020.0-142 · Cited by 15
DIFFBASE: a differential factbase for effective software evolution management
Wu, X., Zhu, C., & Li, Y. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468605 · Cited by 16
FPA-FLPreprint
Feyzi, F., & Parsa, S. A. (2018) · arXiv (Cornell University) · Cited by 9
Do bugs lead to unnaturalness of source code?
Jiang, Y., Liu, H., Zhang, Y., Ji, W., Zhong, H., & Zhang, L. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549149 · Cited by 16
Toward granular search-based automatic unit test case generation
Pecorelli, F., Grano, G., Palomba, F., Gall, H. C., & Lucia, A. D. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10451-x · Cited by 13
Improving Patch Correctness Analysis via Random Testing and Large Language Models
Molina, F., Copia, J. M., & Gorla, A. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00036 · Cited by 18
“We specifically refer to the test that unveils the defect in an incorrect patch, not the conventional fault-revealing test for bugs (as in Defects4J [20]).”
MethodInfluentialPromises and perils of using Transformer-based models for SE research
Xiao, Y., Zuo, X., Lu, X., Dong, J. S., Cao, X., & Beschastnikh, I. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107067 · Cited by 28
SmartFL: Semantics Based Probabilistic Fault Localization
Wu, Y., Liu, Y., Yin, Y., Zeng, M., Ye, Z., Zhang, X., Xiong, Y., & Zhang, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3574487 · Cited by 11
“We take the projects from Defects4J [15] version 2.0 as our benchmark suite.”
InfluentialSteenhoek, B. J., Sivaraman, K., Gonzalez, R. S., Mohylevskyy, Y., Moghaddam, R. Z., & Le, W. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00126 · Cited by 23
AsserT5: Test Assertion Generation Using a Fine-Tuned Code Language Model
Primbs, S., Fein, B., & Fraser, G. (2025) · doi:10.1109/ast66626.2025.00008 · Cited by 17
“1) Experimental Setup: Defects4J [34] is a database and extensible framework that contains bugs that have occurred in real projects.”
From Bugs to Benchmarks: A Comprehensive Survey of Software Defect Datasets
Zhu, H.-N., Furth, R. M., Pradel, M., & Rubio-González, C. (2026) · ACM Computing Surveys · doi:10.1145/3797033 · Cited by 25
“Datasets Count Functional [3, 6, 7, 9, 11, 12, 14, 19, 22, 28, 33, 38, 40, 41, 44, 46, 49, 50, 53, 58–60, 63, 64, 67–70, 74, 79, 83, 85, 87–90, 92, 94, 97, 99, 99, 102, 105, 106, 110, 112, 114–119, 124, 130, 134, 135, 137, 138, 145, 148–150, 153, 154, 163, 164, 167, 168, 171–173, 177, 179–184, 187,…”
InfluentialEdSketch: execution-driven sketching for Java
Hua, J., Zhang, Y., Zhang, Y., & Sarfraz Khurshid, D. (2019) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-019-00512-8 · Cited by 9
Generating Class-Level Integration Tests Using Call Site Information
Derakhshanfar, P., Devroey, X., Panichella, A., Zaidman, A., & Arie Van Deursen, A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3209625 · Cited by 16
Using contextual knowledge in interactive fault localization
Horváth, F., Beszédes, Á., Vancsics, B., Balogh, G., Vidács, L., & Gyimóthy, T. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10190-x · Cited by 12
Graph Neural Network based Two-Phase Fault Localization Approach
Li, Z., Tang, E., Chen, X., Wang, L., & Li, X. (2022) · Asia-Pacific Symposium on Internetware · doi:10.1145/3545258.3545266 · Cited by 8
JUGE: An Infrastructure for Benchmarking Java Unit Test Generators
Devroey, X., Gambi, A., Galeotti, J. P., Just, R., Kifetew, F. M., Panichella, A., & Panichella, S. (2021) · Conicet · doi:10.48550/arxiv.2106.07520 · Cited by 11
Self-citationHow Closely are Common Mutation Operators Coupled to Real Faults?
Gay, G., & Salahirad, A. (2023) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst57152.2023.00021 · Cited by 8
“We broadly adopt the same conventions followed by our experimental subject, Defects4J [15].”
MethodOn Comparing Mutation Testing Tools through Learning-based Mutant Selection
Ojdanić, M., Khanfir, A., Garg, A., Degiovanni, R., Papadakis, M., & Traon, Y. L. (2023) · doi:10.1109/ast58925.2023.00008 · Cited by 18
“We use Defects4J [27] v2.”
Automated Code Editing With Search-Generate-Modify
Liu, C., Cetin, P., Patodia, Y., Ray, B., Chakraborty, S., & Ding, Y. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3376387 · Cited by 18
“Bug Fixing Data: The effectiveness of the pipeline is measured with Defects4j [47] and QuixBugs[48].”
MethodSoftware Analysis for the Web: Achievements and Prospects
Mesbah, A. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.109 · Cited by 8
Using Search-Based Test Generation to Discover Real Faults in Guava
Almulla, H. K., Salahirad, A., & Gay, G. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66299-2_13 · Cited by 8
Too trivial to test? An inverse view on defect prediction to identify methods with low fault risk
Niedermayr, R., Röhm, T., & Wagner, S. (2019) · PeerJ Computer Science · doi:10.7717/peerj-cs.187 · Cited by 7
A Systematic Evaluation of Problematic Tests Generated by EvoSuite
Fan, Z. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00068 · Cited by 8
Do Bug-Fix Types Affect Spectrum-Based Fault Localization Algorithms' Efficiency?
Szatmári, A., Vancsics, B., & Beszédes, Á. (2020) · doi:10.1109/vst50071.2020.9051638 · Cited by 8
Improving the Accuracy of Spectrum-based Fault Localization for Automated Program Repair
Kuma, T., Higo, Y., Matsumoto, S., & Kusumoto, S. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389290 · Cited by 8
Towards a new Test Case Prioritization Approach based on Fuzzy Clustering Analysis
Veșcan, A., & Şerban, C. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00091 · Cited by 8
Automatic Program Repair as Semantic Suggestions: An Empirical Study
Campos, D., Restivo, A., Ferreira, H. S., & Ramos, A. J. M. M. (2021) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst49551.2021.00032 · Cited by 16
Yang, C. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473496 · Cited by 10
Embedding and classifying test execution traces using neural networks
Tsimpourlas, F., Rooijackers, G., Rajan, A., & Allamanis, M. (2021) · IET Software · doi:10.1049/sfw2.12038 · Cited by 12
Wang, P., Brown, C., Jennings, J. A., & Stolee, K. T. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10033-1 · Cited by 11
An Empirical Study on Higher-Order Mutation-Based Fault Localization
Wang, H., Li, Z., Liu, Y., & Chen, X. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500012 · Cited by 8
Is refactoring always a good egg? exploring the interconnection between bugs and refactorings
Bagheri, A., & Hegedűs, P. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528034 · Cited by 12
JTExpert at the fourth unit testing tool competition
Sakti, A., Pesant, G., & Guéhéneuc, Y. (2016) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1145/2897010.2897021 · Cited by 8
An empirical study on the effects of code visibility on program testability
Ma, L., Zhang, C., Yu, B., & Satō, H. (2016) · Software Quality Journal · doi:10.1007/s11219-016-9340-8 · Cited by 8
Learning Fault Localisation for both Humans and Machines Using Multi-objective GP
Choi, K., Sohn, J., & Yoo, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_20 · Cited by 9
Poster: Aiding Java Developers with Interactive Fault Localization in Eclipse IDE
Balogh, G., Horváth, F., & Beszédes, Á. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00045 · Cited by 8
Venugopal, Y., Phung, Q.-N., & Lee, E. (2020) · Applied Sciences · doi:10.3390/app10051593 · Cited by 8
Briem, J. A., Smit, J., Sellik, H., Rapoport, P., Gousios, G., & Aniche, M. F. (2020) · doi:10.1145/3387940.3391464 · Cited by 8
On using Stack Overflow comment-edit pairs to recommend code maintenance changes
Tang, H., & Nadi, S. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09954-8 · Cited by 12
Challenges in Using Search-Based Test Generation to Identify Real Faults in Mockito
Gay, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-47106-8_17 · Cited by 8
Testing Software in Production Environments with Data from the Field
Gazzola, L., Mariani, L., Orrù, M., Pezzè, M., & Tappler, M. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00017 · Cited by 12
The Impact of Rare Failures on Statistical Fault Localization: The Case of the Defects4J Suite
Kucuk, Y., Henderson, T. A. D., & Podgurski, A. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00012 · Cited by 8
Learning a Static Bug Finder from DataPreprint
Wang, Y., Gao, F., Wang, L., & Wang, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05579 · Cited by 13
ReAssert: Deep Learning for Assert GenerationPreprint
White, R., & Krinke, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09784 · Cited by 22
A ground-truth dataset of real security patchesPreprint
Reis, S., & Abreu, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09635 · Cited by 8
Entropy Guided Spectrum Based Bug Localization Using Statistical Language Model
Chakraborty, S., Li, Y., Irvine, M., Saha, R., & Ray, B. (2018) · arXiv · doi:10.48550/arxiv.1802.06947 · Cited by 7
“We used two publicly available bug dataset: Defects4J [32] and ManyBugs [34] (see Table 4).”
MethodTrickyBugs: A Dataset of Corner-case Bugs in Plausible Programs
Liu, K., Han, Y., Liu, Y., Chen, Z., Zhang, J. M., Sarro, F., Huang, G., & Ma, Y. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644870 · Cited by 18
Integrating neural mutation into mutation-based fault localization: A hybrid approach
Liu, H., Li, Z., Han, B., Chen, X., Paul, D., & Liu, Y. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112281 · Cited by 9
Assertions in software testing: survey, landscape, and trends
Taromirad, M., & Runeson, P. (2025) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-025-00794-1 · Cited by 16
“2 Alagarsamy et al. [2] use Defects4J repository [48] to evaluate A3Test’s performance.”
RAG-Driven multiple assertions generation with large language models
Liu, Z., Wang, H., Xu, T., & Wang, B. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10641-1 · Cited by 12
“Tomeasurewhethergeneratedassertionscanassistinexposingbugsinthereal-wordprojects, weconductanexperimentontheDefects4J(Justetal.2014)thatprovidesapopularbench-marktocompile,execute,andevaluatetestcasesonexposingbugs.”
ReAPR: Automatic program repair via retrieval-augmented large language models
Liu, Z., Du, X., & Liu, H. (2025) · Software Quality Journal · doi:10.1007/s11219-025-09728-1 · Cited by 15
“Existing approaches to APR can be categorized as two groups: template-based (Just et al., 2014; Li et al., 2022; Harer et al., 2018; Li et al., 2020; Le et al., 2019) and deep-learning-based (Tan & Roychoudhury, 2015; Chen et al., 2017; Weimer et al., 2009; Long & Rinard, 2015).”
InfluentialWang, B., Deng, M., Chen, M., Lin, Y., Zhou, J., & Zhang, J. M. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00549-x · Cited by 13
“0 (Just et al. 2014), a dataset containing 760 real-world Java bugs from 17 high-impact open-source projects.”
InfluentialUnderstanding LLM-Driven Test Oracle Generation
Bodicoat, A., Jahangirova, G., & Terragni, V. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00011 · Cited by 16
“Similar to established bug benchmarks such as D E - FECTS 4J [22], each repository in the dataset includes a buggy version, where bug-revealing test cases fail, and a corresponding correct version, where the same test cases pass.”
Sohn, J., & Papadakis, M. (2022) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre55969.2022.00023 · Cited by 14
T-Evos: A Large-Scale Longitudinal Study on CI Test Execution and Failure
Chen, A. R., Chen, T.-H. P., & Wang, S. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3218264 · Cited by 12
“Just et al. [9] created the Defects4J benchmark that contains failed test cases caused by real bugs, code coverage of the failing test cases, and the corresponding fixes.”
BackgroundMethodRoslan, M. F., Rojas, J. M., & McMinn, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21251-2_2 · Cited by 12
Code-aware fault localization with pre-training and interpretable machine learning
Zhang, Z., Li, Y., Yang, S., Zhang, Z., & Lei, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121689 · Cited by 11
BugsPHP: A dataset for Automated Program Repair in PHP
Pramod, D., Silva, T. D., Thabrew, U., Shariffdeen, R., & Wickramanayake, S. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644878 · Cited by 7
Papotti, A., Paramitha, R., & Massacci, F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10506-z · Cited by 13
Dependency-Aware Code Naturalness
Yang, C., Chen, J., Jiang, J., & Huang, Y. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689794 · Cited by 11
“For the former, we used all the three datasets that have been used in this application, i.e., Defects4J [Just et al. 2014], GrowingBugs [gro 2023], SmartSHARK [Herbold et al. 2020].”
DRMiner: Extracting Latent Design Rationale from Jira Issue Logs
Zhao, J., Yang, Z., Zhang, L., Lian, X., Yang, D., & Tan, X. L. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695019 · Cited by 21
“• Our evaluation of DR’s utility focuses exclusively on issue resolution requiring changes to no more than 22 lines of code, reflecting the common scale found in the widely-referenced APR dataset, Defects4J [28].”
Automated Program Repair: Emerging Trends Pose and Expose Problems for Benchmarks
Renzullo, J., Reiter, P., Weimer, W. R., & Forrest, S. (2025) · ACM Computing Surveys · doi:10.1145/3704997 · Cited by 25
“When it is included, the reported information varies widely and can include an upper-bound on wall clock time [71] or setting a limit on number of test suite evaluations [50], without reporting efficiency in sufficient detail to allow comparison across papers.”
InfluentialFully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models
Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3719351 · Cited by 17
“…of instruction-driven LLMs has been explored by Fan et al. [2023], where the initial synthesis is done by the Codex model [Chen et al., 2021b] and for program repair both Codex and state-of-the-art specialized repair tools such as TBar and Recoder [Just et al., 2014] are considered and compared.”
RepairBench: Leaderboard of Frontier Models for Program Repair
Silva, A., & Monperrus, M. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00006 · Cited by 31
“RepairBench V1 includes the only two benchmarks that meet all those criteria: Defects4J v2 [Just et al., 2014], a widely-adopted benchmark in software engineering research, contains 835 real-world bugs from 17 open-source Java projects.”
Experimental Analysis of m-ACO Technique for Regression Testing
Solanki, K., Singh, Y., & Dalal, S. (2016) · Indian Journal of Science and Technology · doi:10.17485/ijst/2016/v9i30/86588 · Cited by 7
Automated patch extraction via syntax- and semantics-aware Delta debugging on source code changes
Hashimoto, M., Mori, A., & Izumida, T. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236047 · Cited by 11
A novel approach for mutant diversity-based fault localization: DAM-FL
Gupta, N., Sharma, A., & Pachariya, M. K. (2019) · International Journal of Computers and Applications · doi:10.1080/1206212x.2019.1617504 · Cited by 7
A Large Scale Study On the Effectiveness of Manual and Automatic Unit Test Generation
Souza, B., & Machado, P. D. L. (2020) · Brazilian Symposium on Software Engineering · doi:10.1145/3422392.3422407 · Cited by 9
Assessing test suites of extended finite state machines against model‐ and code‐based faults
El‐Fakih, K., Alzaatreh, A., & Türker, U. C. (2021) · Software Testing Verification and Reliability · doi:10.1002/stvr.1789 · Cited by 8
A dataset of regressions in web applications detected by end-to-end tests
Soto-Sánchez, Ó., Maes-Bermejo, M., Gallego, M., & Gortázar, F. (2021) · Software Quality Journal · doi:10.1007/s11219-021-09566-x · Cited by 9
FDG: a precise measurement of fault diagnosability gain of test cases
An, G., & Yoo, S. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534370 · Cited by 11
CRMF: A fault localization approach based on class reduction and method call frequency
Liu, H., Li, Z., Wang, H., Liu, Y., & Chen, X. (2022) · Software Practice and Experience · doi:10.1002/spe.3178 · Cited by 9
Automated Program Repair from Fuzzing Perspective
Kim, Y., Han, S., Khamit, A. Y., & Yi, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598101 · Cited by 11
“2 [23], considering its widespread use for APR studies [24, 30–33].”
BackgroundMethodThe Future Can’t Help Fix The Past: Assessing Program Repair In The Wild
Kabadi, V., Kong, D., Xie, S., Bao, L., Prana, G. A. A., Le, T.-D. B., Le, X.-B. D., & Lo, D. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00017 · Cited by 10
“To answer this research question, we compute a number of statistics shown in Table VI from failure-exposing test cases of 102 bugs in our dataset and compare with those of 395 bugs from Defects4J [14].”
BackgroundMethodResultInfluentialROBUST: 221 bugs in the Robot Operating System
Timperley, C. S., van der Hoorn, G., SANTOS, A. F. D. M. S., Deshpande, H., & Wąsowski, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10440-0 · Cited by 14
“BugSwarm is a relatively new benchmark that currently consists of over 3000 bug fixes in open-source Java and Python projects.”
BackgroundRunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 36
Towards Addressing the Patch Overfitting Problem
Qi, X. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-c.2017.42 · Cited by 8
Golagha, M., Raisuddin, A. M., Mittag, L., Hellhake, D., & Pretschner, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3183486 · Cited by 7
Adapting Unit Tests by Generating Combinatorial Test Data
Felbinger, H., Wotawa, F., & Nica, M. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00072 · Cited by 7
Derakhshanfar, P., Devroey, X., Zaidman, A., Arie Van Deursen, A., & Panichella, A. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416643 · Cited by 6
Understanding the Non-Repairability Factors of Automated Program Repair Techniques
Lin, B., Wang, S., Wen, M., Zhang, Z., Wu, H., Qint, Y., & Mao, X. (2020) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec51365.2020.00015 · Cited by 7
Expanding Fix Patterns to Enable Automatic Program Repair
Nowack, V., Bowes, D., Counsell, S., Hall, T., Haraldsson, S. O., Winter, E., & Woodward, J. R. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00015 · Cited by 9
Tailored Mutants Fit Bugs BetterPreprint
Allamanis, M., Barr, E. T., Just, R., & Sutton, C. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02516 · Cited by 17
Self-citationFLACOCO: Fault Localization for Java based on Industry-grade CoveragePreprint
Silva, A., Martínez, M., Danglot, B., Ginelli, D., & Monperrus, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12513 · Cited by 24
JLeaks: A Featured Resource Leak Repository Collected From Hundreds of Open-Source Java Projects
Liu, T., Ji, W., Dong, X., Yao, W., Wang, Y., Liu, H., Peng, H., & Wang, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639162 · Cited by 9
“Other defect repositories, e.g, Defects4j [26] and Bears [35], contain different defects with few resource leaks, making them unsuitable for comparative experiments with JLeaks.”
BackgroundExploring Large Language Models in Information Systems: A Survey
Bianchini, F., & Marinacci, M. (2025) · Springer eBooks · doi:10.1007/978-3-031-92285-5_5 · Cited by 9
The Impact of Fine-Tuning Large Language Models on Automated Program Repair
Machacek, R., Grishina, A., Hort, M., & Moonen, L. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00042 · Cited by 15
“Defects4J [60] is a collection of real-world Java bugs collected from multiple open-source projects.”
Enabledness-based Testing of Object Protocols
Godoy, J., Galeotti, J. P., Garbervetsky, D., & Uchitel, S. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3415153 · Cited by 6
Re-visiting the coupling between mutants and real faults with Defects4J 2.0
Laurent, T., Gaffney, S., & Ventresque, A. (2022) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw55395.2022.00042 · Cited by 6
Automatically identifying shared root causes of test breakages in SAP HANA
An, G., Yoon, J., Sohn, J., Hong, J., Hwang, D., & Yoo, S. (2022) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3510457.3513051 · Cited by 6
Cerberus: a Program Repair Framework
Shariffdeen, R., Мирчев, М., Noller, Y., & Roychoudhury, A. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00028 · Cited by 13
“M ANY B UGS [14], Q UIX B UGS [15], B EARS [16], I NTRO C LASS J AVA [17] and D EFECTS 4J [18] are benchmarks consisting of functionality errors reported as test case failures for C/C++ and Java programs.”
Ye, H., Chen, Z., & Goues, C. L. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00163 · Cited by 14
“For instance, even widely-used datasets like Defects4J [1] and ManyBugs [2] are limited to containing fewer than a thousand bugs each.”
Boosting fault localization of statements by combining topic modeling and Ochiai
Vacheret, R., Pérez, F., Ziadi, T., & Hillah, L. M. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107499 · Cited by 7
Detecting, Creating, Repairing, and Understanding Indivisible Multi-Hunk Bugs
Xin, Q., Wu, H., Tang, J., Liu, X., Reiss, S. P., & Xuan, J. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660828 · Cited by 16
“There has been evidence [34, 89] showing that multi-hunk bugs are common.”
InfluentialTowards More Precise Coincidental Correctness Detection With Deep Semantic Learning
Xie, H., Lei, Y., Yan, M. L., Li, S., Mao, X., Yu, Y., & Lo, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3481893 · Cited by 7
“2 [22] dataset with all real faults.”
Enhancing Automated Program Repair with Solution Design
Zhao, J., Yang, D., Zhang, L., Lian, X., Yang, Z., & Liu, F. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695537 · Cited by 40
“A contributing factor is the greater complexity of issues in Solr, as indicated by a higher proportion of patches with changed lines in the ranges of [5,10),[15,20), [20,22), and much less proportion in the range of [1,5), compared to those in Flink (refer to Table 2).”
InfluentialFastFixer: An Efficient and Effective Approach for Repairing Programming Assignments
Liu, F., Liu, Z., Zhao, Q., Jiang, J., Zhang, L., Sun, Z., Li, G., Li, Z., & Ma, Y. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695062 · Cited by 16
“To facilitate further research, the dataset and code scripts are available at https://github.com/LiuFang816/FastFixer.”
The Python Software Quality Dataset
Moldovan, V., Berciu, L.-M., & Patcas, R.-D. (2024) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa64295.2024.00066 · Cited by 6
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.