Defects4J: a database of existing faults to enable controlled testing studies for Java programs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Empirical studies in software testing research may not be comparable, reproducible, or characteristic of practice. One reason is that real bugs are too infrequently used in software testing research. Extracting and reproducing real bugs is challenging and as a result hand-seeded faults or mutants are commonly used as a substitute. This paper presents Defects4J, a database and extensible framework providing real bugs to enable reproducible studies in software testing research. The initial version of Defects4J contains 357 real bugs from 5 real-world open source pro- grams. Each real bug is accompanied by a comprehensive test suite that can expose (demonstrate) that bug. Defects4J is extensible and builds on top of each program’s version con- trol system. Once a program is configured in Defects4J, new bugs can be added to the database with little or no effort. Defects4J features a framework to easily access faulty and fixed program versions and corresponding test suites. This framework also provides a high-level interface to common tasks in software testing research, making it easy to con- duct and reproduce empirical studies. Defects4J is publicly available at http://defects4j.org.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [21, 23].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [21]
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [13]
In the citing paper
“Martinez et al. [40] manually assessed 84 plausible patches generated on the Defects4J benchmark [30], and found that only 11 of them are correct.”
From Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study (2026) · cites this paper as [30]
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [84]
Citations
Cites
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationExtraction of bug localization benchmarks from history
Dallmeier, V., & Zimmermann, T. (2007) · Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering · doi:10.1145/1321631.1321702 · Cited by 216
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citation
Cited by
Items 101–200 of 1,012
A3Test: Assertion-Augmented Automated Test case generation
Alagarsamy, S., Tantithamthavorn, C., & Aleti, A. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107565 · Cited by 90
BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes
Tomassi, D. A., Dmeiri, N., Wang, Y., Bhowmick, A., Liu, Y.-C., Devanbu, P. T., Vasilescu, B., & Rubio-Gonzalez, C. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00048 · Cited by 88
SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics
Ye, H., Martinez, M., Luo, X., Zhang, T., & Monperrus, M. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3556926 · Cited by 86
How Effective Are Code Coverage Criteria?
Hemmati, H. (2015) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2015.30 · Cited by 87
F a C o Y: a code-to-code search engine
Kim, K., Kim, D., Bissyandé, T. F., Choi, E., Li, L., Klein, J., & Traon, Y. L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180187 · Cited by 117
A Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150
To Type or Not to Type: Quantifying Detectable Bugs in JavaScript
Gao, Z., Bird, C., & Barr, E. T. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.75 · Cited by 124
Comparing and combining test-suite reduction and regression test selection
Shi, A., Yung, T., Gyori, A., & Marinov, D. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2786878 · Cited by 104
On Reliability of Patch Correctness Assessment
Le, X.-B. D., Bao, L., Lo, D., Xia, X., Li, S., & Păsăreanu, C. S. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00064 · Cited by 121
An Empirical Study of Boosting Spectrum-Based Fault Localization via PageRank
Zhang, M., Li, Y., Li, X., Chen, L., Zhang, Y., Zhang, L., & Khurshid, S. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2911283 · Cited by 84
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
Yasunaga, M., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10636 · Cited by 81
“Another line of work learns from labeled datasets of how programmers edit code (e.g. error resolution records) (Just et al., 2014; Chen et al., 2019).”
BackgroundAmann, S., Nadi, S., Nguyen, H. A., Nguyen, T. N., & Mezini, M. (2016) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2901739.2903506 · Cited by 80
Keller, F., Grunske, L., Heiden, S., Filieri, A., van Hoorn, A., & Lo, D. (2017) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2017.22 · Cited by 83
Yu, X. L., Al-Bataineh, O. I., Lo, D., & Roychoudhury, A. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3402450 · Cited by 88
A Survey of Dynamic Analysis and Test Generation for JavaScript
Andreasen, E., Gong, L., Møller, A., Pradel, M., Selakovic, M., Sen, K., & Staicu, C.-A. (2017) · ACM Computing Surveys · doi:10.1145/3106739 · Cited by 92
Automated patch correctness assessment: how far are we?
Wang, S., Wen, M., Lin, B., Wu, H., Qin, Y., Zou, D., Mao, X., & Jin, H. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416590 · Cited by 79
A critical review on the evaluation of automated program repair systems
Liu, K., Li, L., Koyuncu, A., Kim, D., Liu, Z., Klein, J., & Bissyandé, T. F. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110817 · Cited by 79
“Table 1 provides an example of assessment results excerpted from the paper describing Hercules (Saha et al., 2019), one of the most recent state-of-the-art works on APR that was tested on the Defects4J (Just et al., 2014) programs.”
BackgroundMethodInfluentialInferring Program Transformations From Singular Examples via Big Code
Jiang, J., Ren, L., Xiong, Y., & Zhang, L. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00033 · Cited by 73
Selecting fault revealing mutants
Chekam, T. T., Papadakis, M., Bissyandé, T. F., Traon, Y. L., & Sen, K. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09778-7 · Cited by 98
Unit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 346
Automated patch assessment for program repair at scale
Ye, H., Martínez, M., & Monperrus, M. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09920-w · Cited by 114
Learning How to Mutate Source Code from Bug-Fixes
Tufano, M., Watson, C., Bavota, G., Di Penta, M., White, M., & Poshyvanyk, D. (2019) · 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2019.00046 · Cited by 69
Kintis, M., Papadakis, M., Papadopoulos, A., Valvis, E., Malevris, N., & Traon, Y. L. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9582-5 · Cited by 89
Explainable automated debugging via large language model-driven scientific debugging
Kang, S., Chen, B., Yoo, S., & Lou, J.-G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10594-x · Cited by 68
“2 and 2.0 benchmarks [15] consisting of real-world bugs.”
InfluentialGRT: Program-Analysis-Guided Random Testing (T)
Ma, L., Artho, C., Zhang, C., Sato, H., Gmeiner, J., & Ramler, R. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.49 · Cited by 71
Overfitting in semantics-based automated program repair
Le, X.-B. D., Thung, F., Lo, D., & Goues, C. L. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9577-2 · Cited by 77
Crashing Simulated Planes is Cheap: Can Simulation Detect Robotics Bugs Early?
Timperley, C. S., Afzal, A., Katz, D. S., Hernandez, J. M., & Goues, C. L. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00040 · Cited by 89
A Test-Suite Diagnosability Metric for Spectrum-Based Fault Localization Approaches
Perez, A., Abreu, R., & Arie Van Deursen, A. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.66 · Cited by 87
LSRepair: Live Search of Fix Ingredients for Automated Program Repair
Liu, K., Koyuncu, A., Kim, K., Kim, D., & F. Bissyande, T. (2018) · 2018 25th Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2018.00085 · Cited by 67
Automated Classification of Overfitting Patches With Statically Extracted Code Features
Ye, H., Gu, J. N., Martínez, M., Durieux, T., & Monperrus, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3071750 · Cited by 96
Campos, J., & Souto, A. (2021) · IEEE/ACM International Workshop on Quantum Software Engineering (Q-SE) · doi:10.1109/q-se52541.2021.00013 · Cited by 81
A Survey of Challenges in Spectrum-Based Software Fault Localization
Sarhan, Q. I., & Beszédes, Á. (2022) · IEEE Access · doi:10.1109/access.2022.3144079 · Cited by 95
What It Would Take to Use Mutation Testing in Industry—A Study at Facebook
Beller, M., Wong, C.-P., Bader, J., Scott, A., Machalica, M., Chandra, S., & Meijer, E. (2021) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip52600.2021.00036 · Cited by 68
ThinkRepair: Self-Directed Automated Program Repair
Yin, X., Ni, C., Wang, S., Li, Z., Zeng, L., & Yang, X. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680359 · Cited by 68
“We conduct experiments on two widely studied dataset (i.e., De-fects4J [22] and QuixBugs [32]) by comparing ThinkRepair with 12 state-of-the-art APR approaches.”
Does Mutation Testing Improve Testing Practices?
Petrović, G., Ivanković, M., Fraser, G., & Just, R. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00087 · Cited by 67
Self-citationSpectrum-based Software Fault Localization: A Survey of Techniques, Advances, and ChallengesPreprint
de Souza, H. A., Chaim, M. L., & Kon, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.04347 · Cited by 126
SeqTrans: Automatic Vulnerability Fix Via Sequence to Sequence Learning
Chi, J., Qu, Y., Liu, T., Zheng, Q., & Yin, H. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3156637 · Cited by 95
Exploring and exploiting the correlations between bug-inducing and bug-fixing commits
Wen, M., Wu, R., Liu, Y., Tian, Y., Xie, X., Cheung, S.-C., & Su, Z. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338962 · Cited by 64
A study of effectiveness of deep learning in locating real faults
Zhang, Z., Lei, Y., Mao, X., Yan, M. L., Xu, L., & Zhang, X. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106486 · Cited by 68
Combining Spectrum-Based Fault Localization and Statistical Debugging: An Empirical Study
Jiang, J., Wang, R., Xiong, Y., Chen, X., & Zhang, L. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00054 · Cited by 55
Context-Aware Code Change Embedding for Better Patch Correctness Assessment
Lin, B., Wang, S., Wen, M., & Mao, X. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3505247 · Cited by 88
CIRCLE: continual repair across programming languages
Yuan, W., Zhang, Q., He, T., Fang, C., Hung, N. Q. V., Hao, X., & Yin, H. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534219 · Cited by 62
SpecGen: Automated Generation of Formal Program Specifications via Large Language Models
Ma, L.-Z., Liu, S.-Q., Li, Y., Xie, X.-F., & Bu, L. (2024) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00129 · Cited by 128
“To mitigate it, we refer to Xia et al. [51] to design the prompt.”
Combining symbolic execution and search-based testing for programs with complex heap inputs
Braione, P., Denaro, G., Mattavelli, A., & Pezzè, M. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092715 · Cited by 71
A large scale empirical comparison of state-of-the-art search-based test case generators
Panichella, A., Kifetew, F. M., & Tonella, P. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.08.009 · Cited by 80
Learning semantic program embeddings with graph interval neural network
Wang, Y., Wang, K., Gao, F., & Wang, L. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428205 · Cited by 77
Scalable Approaches for Test Suite Reduction
Cruciani, E., Miranda, B., Verdecchia, R., & Bertolino, A. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00055 · Cited by 68
Gao, X. Y., Mechtaev, S., & Roychoudhury, A. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330558 · Cited by 81
Phoenix: automated data-driven synthesis of repairs for static analysis violations
Bavishi, R., Yoshida, H., & Prasad, M. R. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338952 · Cited by 81
Inferring mutant utility from program context
Just, R., Kurtz, B., & Ammann, P. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092732 · Cited by 60
Self-citationYu, Z., Martínez, M., Danglot, B., Durieux, T., & Monperrus, M. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9619-4 · Cited by 106
An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions
Petrović, G., Ivanković, M., Kurtz, B., Ammann, P., & Just, R. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00027 · Cited by 58
Self-citationMitigating the effects of flaky tests on mutation testing
Shi, A., Bell, J., & Marinov, D. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330568 · Cited by 58
Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?
Endres, M., Fakhoury, S., Chakraborty, S., & Lahiri, S. K. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660791 · Cited by 188
“We evaluate nl2postcond ’s bug-catching potential using Defects4J [20], a benchmark of historical Java bugs.”
InfluentialA Closer Look at Real-World Patches
Liu, K., Kim, D., Koyuncu, A., Li, L., Bissyandé, T. F., & Traon, Y. L. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00037 · Cited by 57
Improving Fault Localization by Integrating Value and Predicate Based Causal Inference Techniques
Kucuk, Y., Henderson, T. A. D., & Podgurski, A. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00066 · Cited by 76
KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair
Jiang, N., Lutellier, T., Lou, Y., Tan, L., Goldwasser, D., & Zhang, X. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00111 · Cited by 101
AI for DevSecOps: A Landscape and Future Opportunities
Fu, M. C., Pasuksmit, J., & Tantithamthavorn, C. K. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712190 · Cited by 110
A Survey on Machine Learning Techniques for Source Code AnalysisPreprint
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R. K., Vats, I., Moazen, H., & Sarro, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09610 · Cited by 56
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
“However, none of the existing datasets (Just et al., 2014; Karampatsis & Sutton, 2019) present code context at the scale of SWE-bench.”
BackgroundRevisiting the relationship between fault detection, test adequacy criteria, and test set size
Chen, Y. T., Gopinath, R., Tadakamalla, A., Ernst, M. D., Holmes, R., Fraser, G., Ammann, P., & Just, R. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416667 · Cited by 65
Self-citationDo automated program repair techniques repair hard and important bugs?
Motwani, M., Sankaranarayanan, S., Just, R., & Brun, Y. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9550-0 · Cited by 63
Self-citationGamma: Revisiting Template-Based Automated Program Repair Via Mask Prediction
Zhang, Q., Fang, C., Zhang, T., Yu, B., Sun, W., & Chen, Z. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00063 · Cited by 126
“2 [48] in the APR community.”
A public unified bug dataset for java and its assessment regarding metrics and bug prediction
Ferenć, R., Tóth, Z., Ladányi, G., Siket, I., & Gyimóthy, T. (2020) · Software Quality Journal · doi:10.1007/s11219-020-09515-0 · Cited by 68
Empirical evaluation of mutation‐based test case prioritization techniques
Shin, D., Yoo, S., Papadakis, M., & Bae, D. (2018) · Software Testing Verification and Reliability · doi:10.1002/stvr.1695 · Cited by 53
Why My App Crashes? Understanding and Benchmarking Framework-Specific Exceptions of Android Apps
Su, T., Fan, L., Chen, S., Liu, Y., Xu, L., Pu, G., & Su, Z. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3013438 · Cited by 52
Bugs4Q: A Benchmark of Real Bugs for Quantum Programs
Zhao, P., Zhao, J., Miao, Z., & Lan, S. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678908 · Cited by 52
An Empirical Study on the Use of Defect Prediction for Test Case Prioritization
Paterson, D., Campos, J., Abreu, R., Kapfhammer, G. M., Fraser, G., & McMinn, P. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00041 · Cited by 52
Precise Learn-to-Rank Fault Localization Using Dynamic and Static Features of Target Programs
Kim, Y., Mun, S., Yoo, S., & Kim, M. (2019) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3345628 · Cited by 64
Incorporating fault-proneness estimations into coverage-based test case prioritization methods
Mahdieh, M., Mirian‐Hosseinabadi, S., Etemadi, K., Nosrati, A., & Jalali, S. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106269 · Cited by 52
ModelSet: a dataset for machine learning in model-driven engineering
López, J. A. H., Izquierdo, J. L., & Cuadrado, J. S. (2021) · Software & Systems Modeling · doi:10.1007/s10270-021-00929-3 · Cited by 63
Using bug descriptions to reformulate queries during text-retrieval-based bug localization
Chaparro, O., Florez, J. M., & Marcus, A. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9672-z · Cited by 59
Evaluating SZZ Implementations Through a Developer-Informed Oracle
Rosa, G., Pascarella, L., Scalabrino, S., Tufano, R., Bavota, G., Lanza, M., & Oliveto, R. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00049 · Cited by 84
A fine-grained data set and analysis of tangling in bug fixing commits
Herbold, S., Trautsch, A., Ledel, B., Aghamohammadi, A., Ghaleb, T. A., Chahal, K. K., Bossenmaier, T., Nagaria, B., Makedonski, P., Ahmadabadi, M. N., et al. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10083-5 · Cited by 101
An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems
Heiden, S., Grunske, L., Kehrer, T., Keller, F., van Hoorn, A., Filieri, A., & Lo, D. (2019) · Software Practice and Experience · doi:10.1002/spe.2703 · Cited by 51
Quality of Automated Program Repair on Real-World Defects
Motwani, M., Soto, M., Brun, Y., Just, R., & Goues, C. L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2998785 · Cited by 69
Self-citationEmpirically revisiting and enhancing IR-based test-case prioritization
Peng, Q., Shi, A., & Zhang, L. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397383 · Cited by 63
VarFix: balancing edit expressiveness and search effectiveness in automated program repair
Wong, C.-P., Santiesteban, P., Kästner, C., & Goues, C. L. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468600 · Cited by 68
On Multi-Modal Learning of Editing Source Code
Chakraborty, S., & Ray, B. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678559 · Cited by 47
ITER: Iterative Neural Repair for Multi-Location Patches
Ye, H., & Monperrus, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623337 · Cited by 93
“In RQ1, we compare ITER against the state-of-the-art program repair baselines on 10 open-source projects from widely used benchmark Defects4J 2.0 [19].”
MethodInfluentialRepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair
Silva, A., Fang, S., & Monperrus, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3581062 · Cited by 156
“To mitigate the first threat, we evaluate on three benchmarks, including well-established Defects4J [22].”
A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark
Ye, H., Martínez, M., & Monperrus, M. (n.d.) · DiVA (Digitala Vetenskapliga Arkivet) · Cited by 50
Assessing Test Case Prioritization on Real Faults and Mutants
Luo, Q., Moran, K., Poshyvanyk, D., & Penta, M. D. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00033 · Cited by 47
Relation-based test case prioritization for regression testing
Chi, J., Qu, Y., Zheng, Q., Yang, Z., Jin, W., Cui, D., & Liu, T. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110539 · Cited by 47
Java Unit Testing Tool Competition - Seventh Round
Kifetew, F. M., Devroey, X., & Rueda, U. (2019) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1109/sbst.2019.00014 · Cited by 50
Using Observed Behavior to Reformulate Queries during Text Retrieval-based Bug Localization
Chaparro, O., Florez, J. M., & Marcus, A. (2017) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2017.100 · Cited by 61
A Framework for Checking Regression Test Selection Tools
Zhu, C., Legunsen, O., Shi, A., & Gligoric, M. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00056 · Cited by 56
Tihanyi, N., Charalambous, Y., Jain, R., Ferrag, M. A., & Cordeiro, L. C. (2025) · 2025 IEEE/ACM International Conference on Automation of Software Test (AST) · doi:10.1109/ast66626.2025.00020 · Cited by 40
Private API Access and Functional Mocking in Automated Unit Test Generation
Arcuri, A., Fraser, G., & Just, R. (2017) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2017.19 · Cited by 54
Self-citationLi, X., Zhu, S., d’Amorim, M., & Orso, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180242 · Cited by 43
Automated model repair for Alloy
Wang, K., Sullivan, A., & Sarfraz Khurshid, D. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238162 · Cited by 46
Fault localization via efficient probabilistic modeling of program semantics
Zeng, M., Wu, Y., Ye, Z., Xiong, Y., Zhang, X., & Zhang, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510073 · Cited by 50
Fine-tuning spectrum based fault localisation with frequent method item sets
Laghari, G., Murgia, A., & Demeyer, S. (2016) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2970276.2970308 · Cited by 41
Localizing multiple software faults based on evolution algorithm
Zheng, Y., Wang, Z., Fan, X., Chen, X., & Yang, Z. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.02.001 · Cited by 45
VFix: Value-Flow-Guided Precise Program Repair for Null Pointer Dereferences
Xu, X., Sui, Y., Yan, H., & Xue, J. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00063 · Cited by 62
Tufano, M., Deng, S. K., Sundaresan, N., & Svyatkovskiy, A. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528009 · Cited by 51
Learning to Construct Better Mutation Faults
Tian, Z., Chen, J., Zhu, Q., Yang, J., & Zhang, L. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556949 · Cited by 48
A survey on machine learning techniques applied to source code
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R., Vats, I., Moazen, H., & Sarro, F. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111934 · Cited by 173
Causal testing: understanding defects' root causes
Johnson, B., Brun, Y., & Meliou, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380377 · Cited by 78
On the feasibility of automated prediction of bug and non-bug issues
Herbold, S., Trautsch, A., & Trautsch, F. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09885-w · Cited by 62
Metadata
- Authors
- Just, R., Jalali, D., & Ernst, M. D.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2014 International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2014-07-21
- Pages
- 437-440
- OpenAlex
- W2156723666
- Cited by
- 1,447
- References
- 8
- Citations by year
- 2026: 197 · 2025: 201 · 2024: 172 · 2023: 133 · 2022: 128 · 2021: 132 · 2020: 127 · 2019: 139 · 2018: 106 · 2017: 60 · 2016: 34 · 2015: 17 · 2014: 1
- FWCI
- 24.71
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Software Testing and Analysis (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- Defects4J, Java programs, version control, open source software, empirical software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.