BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies
Open accessAbstract
The 2019 edition of Stack Overflow developer survey highlights that, for the first time, Python outperformed Java in terms of popularity. The gap between Python and Java further widened in the 2020 edition of the survey. Unfortunately, despite the rapid increase in Python's popularity, there are not many testing and debugging tools that are designed for Python. This is in stark contrast with the abundance of testing and debugging tools for Java. Thus, there is a need to push research on tools that can help Python developers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
We evaluated our approach on Defects4J and BugsInPy, two widely used benchmarks containing diverse real-world faults in Java and Python projects.
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [35]
The Abstract describes BugsInPy as a benchmark containing 493 real bugs from 17 real-world Python programs, and the Introduction says those projects span diverse domains. The paper describes BugsInPy as newly created and hopes it will support future research; it does not establish that BugsInPy was already widely used. The Introduction identifies Defects4J, rather than BugsInPy, as having been used by hundreds of studies. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Matplotlib: a 2D graphics environment
Hunter, J. D. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.55 · Cited by 39,249 · see citing works
Reback, J., McKinney, W., Bossche, J. V. D., Jbrockmendel, Augspurger, T., Cloud, P., Gfyoung, Sinhrks, Klein, A., Tratner, J., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3509134 · Cited by 4,025 · see citing works
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447 · see citing works
Do, H., Elbaum, S. G., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,161 · see citing works
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 783 · see citing works
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635929 · Cited by 602 · see citing works
Honnibal, M., Montani, I., Honnibal, M., Peters, H., Samsonov, M., Geovedi, J., Regan, J., Orosz, G., Kristiansen, S. L., R., et al. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1212303 · Cited by 543 · see citing works
Le, X.-B. D., Lo, D., & Goues, C. L. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 341 · see citing works
Self-citationPrecise Condition Synthesis for Program Repair
Xiong, Y., Wang, J., Yan, R., Zhang, J., Shi, H., Huang, G., & Zhang, L. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.45 · Cited by 308 · see citing works
Automatic repair of real bugs in java: a large-scale experiment on the defects4j dataset
Martínez, M., Durieux, T., Sommerard, R., Xuan, J., & Monperrus, M. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9470-4 · Cited by 249 · see citing works
FLUCCS: using code and change metrics to improve fault localization
Sohn, J., & Yoo, S. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092717 · Cited by 243 · see citing works
QuixBugs: a multi-lingual program repair benchmark set based on the quixey challenge
Lin, D., Koppel, J., Chen, A., & Solar-Lezama, A. (2017) · Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity · doi:10.1145/3135932.3135941 · Cited by 242 · see citing works
A learning-to-rank based fault localization approach using likely invariants
Le, T.-D. B., Lo, D., Goues, C. L., & Grunske, L. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931049 · Cited by 206 · see citing works
Self-citationSaha, R. K., Lyu, Y., Lam, W., Yoshida, H., & Prasad, M. R. (2018) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3196398.3196473 · Cited by 168 · see citing works
How does regression test prioritization perform in real-world software evolution?
Lu, Y., Lou, Y., Cheng, S., Zhang, L., Hao, D., Zhou, Y., & Zhang, L. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884874 · Cited by 110 · see citing works
Xia, X., Bao, L., Lo, D., & Li, S. (2016) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2016.67 · Cited by 104 · see citing works
Self-citationBugsJS: a Benchmark of JavaScript Bugs
Gyimesi, P., Vancsics, B., Stocco, A., Mazinanian, D., Beszédes, Á., Ferenć, R., & Mesbah, A. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00019 · Cited by 95 · see citing works
BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes
Tomassi, D. A., Dmeiri, N., Wang, Y., Bhowmick, A., Liu, Y.-C., Dévanbu, P., Vasilescu, B., & Rubio-González, C. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00048 · Cited by 91 · see citing works
GRT: Program-Analysis-Guided Random Testing (T)
Ma, L., Artho, C. V., Zhang, C., Satō, H., Gmeiner, J., & Ramler, R. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.49 · Cited by 72 · see citing works
Defexts: A Curated Dataset of Reproducible Real-World Bugs for Modern JVM Languages
Benton, S., Ghanbari, A., & Zhang, L. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00035 · Cited by 23 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Casper O da Costa-Luis. 2019. tqdm: A fast, extensible progress meter for python and cli. Journal of Open Source Software 4, 37 (2019), 1277.
- Thomas Durieux and Rui Abreu. 2019. Critical Review of BugSwarm for Fault Localization and Program Repair. arXiv preprint arXiv:1905.09375 (2019). · arXiv
- Ripon K Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul R Prasad. 2018. Bugs. jar: a large-scale, diverse dataset of real-world java bugs. In Proceedings of the 15th International Conference on Mining Software Repositories. 10-13.
Cited by
Sorted by citations
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · see citing works
Large Language Models for Test-Free Fault Localization
Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623342 · Cited by 144 · see citing works
A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
Kang, S., An, G., & Yoo, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660771 · Cited by 128 · see citing works
Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM
Ryan, G., Jain, S., Shang, M., Wang, S., Ma, X., Ramanathan, M. K., & Ray, B. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643769 · Cited by 125 · see citing works
An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Meng, X., Zhang, J., Liu, Y., Wang, W., Li, S., & Zhang, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00181 · Cited by 113 · see citing works
A Survey on Software Defect Prediction Using Deep Learning
Akimova, E. N., Bersenev, A. Y., Deikov, A. A., Kobylkin, K. S., Konygin, A. V., Mezentsev, I. P., & Misilov, V. E. (2021) · Mathematics · doi:10.3390/math9111180 · Cited by 118 · see citing works
Explainable automated debugging via large language model-driven scientific debugging
Kang, S., Chen, B., Yoo, S., & Lou, J. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10594-x · Cited by 72 · see citing works
A Survey of Challenges in Spectrum-Based Software Fault Localization
Sarhan, Q. I., & Beszédes, Á. (2022) · IEEE Access · doi:10.1109/access.2022.3144079 · Cited by 69 · see citing works
An empirical study of automated unit test generation for Python
Lukasczyk, S., Kroiß, F., & Fraser, G. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10248-w · Cited by 66 · see citing works
Fixing dependency errors for Python build reproducibility
Mukherjee, S., Almanza, A., & Rubio-González, C. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464797 · Cited by 62 · see citing works
Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting
Li, T.-O., Zong, W., Wang, Y., Tian, H., Wang, Y., Cheung, S.-C., & Kramer, J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00089 · Cited by 70 · see citing works
CoverUp: Effective High Coverage Test Generation for Python
Pizzorno, J. A., & Berger, E. D. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729398 · Cited by 84 · see citing works
Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors
Peng, Y., Gao, S., Gao, C., Huo, Y., & Lyu, M. R.-T. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608132 · Cited by 53 · see citing works
Evaluating large language models for software testing
Li, Y., Liu, P., Wang, H., Chu, J., & Wong, W. E. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103942 · Cited by 42 · see citing works
Test Oracle Automation in the Era of LLMs
Molina, F., Gorla, A., & d’Amorim, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715107 · Cited by 41 · see citing works
You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
Bouzenia, I., & Pradel, M. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3728922 · Cited by 35 · see citing works
Comprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Zhang, J., Bao, X., Wang, X., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3532759 · Cited by 30 · see citing works
Demystifying Faulty Code: Step-by-Step Reasoning for Explainable Fault Localization
Widyasari, R., Ang, J. W., Nguyen, T. G., Sharma, N., & Lo, D. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00064 · Cited by 29 · see citing works
Self-citationWidyasari, R., Prana, G. A. A., Haryono, S. A., Wang, S., & Lo, D. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10189-4 · Cited by 25 · see citing works
Self-citationFault localization to detect co-change fixing locations
Li, Y., Wang, S., & Nguyen, T. N. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549137 · Cited by 25 · see citing works
Bugs in machine learning-based systems: a faultload benchmark
Morovati, M. M., Nikanjam, A., Khomh, F., & Jiang, Z. M. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10291-1 · Cited by 23 · see citing works
PyTER: effective program repair for Python type errors
Oh, W.-S., & Oh, H. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549130 · Cited by 22 · see citing works
Out of Context: How important is Local Context in Neural Program Repair?
Prenner, J. A., & Robbes, R. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639086 · Cited by 22 · see citing works
The Fact Selection Problem in LLM-Based Program Repair
Parasaram, N., Yan, H., Yang, B., Flahy, Z., Qudsi, A., Ziaber, D., Barr, E. T., & Mechtaev, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00162 · Cited by 20 · see citing works
Mining Python fix patterns via analyzing fine-grained source code changes
Yang, Y., He, T., Feng, Y., Liu, S., & Xu, B. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10087-1 · Cited by 19 · see citing works
A systematic literature review on benchmarks for evaluating debugging approaches
Hirsch, T., & Hofer, B. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111423 · Cited by 19 · see citing works
Towards understanding bugs in Python interpreters
Liu, D., Feng, Y., Yan, Y., & Xu, B. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10239-x · Cited by 18 · see citing works
The GitHub Recent Bugs Dataset for Evaluating LLM-Based Debugging Applications
Lee, J. Y., Kang, S., Yoon, J., & Yoo, S. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00049 · Cited by 18 · see citing works
Are Large Language Models Memorizing Bug Benchmarks?
Ramos, D., Mamede, C., Jain, K., Canelas, P., Gamboa, C., & Goues, C. L. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00005 · Cited by 16 · see citing works
Optimizing Search-Based Unit Test Generation with Large Language Models: An Empirical Study
Xiao, D., Guo, Y., Li, Y., & Chen, L. (2024) · Asia-Pacific Symposium on Internetware · doi:10.1145/3671016.3674813 · Cited by 15 · see citing works
When Deep Learning Meets Information Retrieval-based Bug Localization: A Survey
Niu, F., Li, C., Liu, K., Xia, X., & Lo, D. (2025) · ACM Computing Surveys · doi:10.1145/3734217 · Cited by 17 · see citing works
Self-citationAkimova, E. N., Bersenev, A. Y., Deikov, A. A., Kobylkin, K. S., Konygin, A. V., Mezentsev, I. P., & Misilov, V. E. (2021) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec53868.2021.00022 · Cited by 15 · see citing works
Learning Realistic Mutations: Bug Creation for Neural Bug Detectors
Richter, C., & Wehrheim, H. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00027 · Cited by 14 · see citing works
Eberlein, M., Smytzek, M., Steinhöfel, D., Grunske, L., & Zeller, A. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616296 · Cited by 14 · see citing works
Tests4Py: A Benchmark for System Testing
Smytzek, M., Eberlein, M., Serçe, B., Grunske, L., & Zeller, A. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663798 · Cited by 13 · see citing works
Denchmark: A Bug Benchmark of Deep Learning-related Software
Kim, M., Kim, Y., & Lee, E. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00070 · Cited by 13 · see citing works
Zhang, J., Hu, X., Xia, X., Cheung, S.-C., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3745765 · Cited by 13 · see citing works
Fine-grained, accurate and scalable source differencing
Falleri, J., & Martínez, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639148 · Cited by 12 · see citing works
An empirical study of fault localization in Python programs
Rezaalipour, M., & Furia, C. A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10475-3 · Cited by 12 · see citing works
Rete: Learning Namespace Representation for Program Repair
Parasaram, N., Barr, E. T., & Mechtaev, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00112 · Cited by 12 · see citing works
BUGSC++: A Highly Usable Real World Defect Benchmark for C/C++
An, G., Kwon, M., Choi, K., Yi, J., & Yoo, S. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00208 · Cited by 11 · see citing works
DyPyBench: A Benchmark of Executable Python Software
Bouzenia, I., Krishan, B. P., & Pradel, M. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643742 · Cited by 11 · see citing works
T5APR: Empowering automated program repair across languages through checkpoint ensemble
Gharibi, R., Sadreddini, M. H., & Fakhrahmad, S. M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112083 · Cited by 13 · see citing works
Neural-MBFL: Improving Mutation-Based Fault Localization by Neural Mutation
Du, B., Han, B., Liu, H., Chang, Z., Liu, Y., & Chen, X. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00168 · Cited by 11 · see citing works
ChatDL: An LLM-Based Defect Localization Approach for Software in IIoT Flexible Manufacturing
Yang, H., Zhou, Y. L., Liang, T., & Kuang, L. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3531512 · Cited by 11 · see citing works
SFLKit: a workbench for statistical fault localization
Smytzek, M., & Zeller, A. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558915 · Cited by 11 · see citing works
VALIDATE: A deep dive into vulnerability prediction datasets
Esposito, M., & Falessi, D. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107448 · Cited by 10 · see citing works
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 24 · see citing works
A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing Techniques
Elgendy, I. T., Hierons, R. M., & McMinn, P. (2025) · Software Testing Verification and Reliability · doi:10.1002/stvr.1914 · Cited by 9 · see citing works
Reflective Unit Test Generation for Precise Type Error Detection with Large Language Models
Yang, C., Wang, Z., Jiang, Y., Yang, L., Zheng, Y., Zhou, J., & Chen, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00233 · Cited by 9 · see citing works
How Execution Features Relate to Failures: An Empirical Study and Diagnosis Approach
Smytzek, M., Eberlein, M., Grunske, L., & Zeller, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3783989 · Cited by 8 · see citing works
From Bugs to Benchmarks: A Comprehensive Survey of Software Defect Datasets
Zhu, H., Furth, R. M., Pradel, M., & Rubio-González, C. (2026) · ACM Computing Surveys · doi:10.1145/3797033 · Cited by 8 · see citing works
Demystifying regular expression bugs
Wang, P., Brown, C., Jennings, J. A., & Stolee, K. T. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10033-1 · Cited by 8 · see citing works
AWSomePy: A Dataset and Characterization of Serverless Applications
Raffa, G., Blasco, J., O’Keeffe, D., & Dash, S. K. (2023) · Workshop on SErverless Systems, Applications and MEthodologies (SESAME) · doi:10.1145/3592533.3592811 · Cited by 8 · see citing works
Integrating neural mutation into mutation-based fault localization: A hybrid approach
Liu, H., Li, Z., Han, B., Chen, X., Paul, D., & Liu, Y. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112281 · Cited by 8 · see citing works
Tanzil, M. H., Chowdhury, S., Modaberi, S., Uddin, G., & Hemmati, H. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112405 · Cited by 7 · see citing works
BugsPHP: A dataset for Automated Program Repair in PHP
Pramod, D., Silva, T. D., Thabrew, U., Shariffdeen, R., & Wickramanayake, S. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644878 · Cited by 7 · see citing works
A dataset of regressions in web applications detected by end-to-end tests
Soto-Sánchez, Ó., Maes-Bermejo, M., Gallego, M., & Gortázar, F. (2021) · Software Quality Journal · doi:10.1007/s11219-021-09566-x · Cited by 7 · see citing works
ROBUST: 221 bugs in the Robot Operating System
Timperley, C. S., van der Hoorn, G., SANTOS, A. F. D. M. S., Deshpande, H., & Wąsowski, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10440-0 · Cited by 7 · see citing works
RunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 11 · see citing works
Towards Effective Static Type-Error Detection for Python
Oh, W., & Oh, H. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695545 · Cited by 6 · see citing works
Yin, X., Ni, C., Li, X., Chen, L., Ma, G., & Yang, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00058 · Cited by 10 · see citing works
How Well Static Type Checkers Work with Gradual Typing? A Case Study on Python
Xu, W., Chen, L., Su, C., Guo, Y., Li, Y., Zhou, Y., & Xu, B. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00039 · Cited by 6 · see citing works
Ye, H., Chen, Z., & Goues, C. L. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00163 · Cited by 6 · see citing works
PyBugHive: A Comprehensive Database of Manually Validated, Reproducible Python Bugs
Antal, G., Vándor, N., Kolláth, I., Mosolygó, B., Hegedűs, P., & Ferenć, R. (2024) · IEEE Access · doi:10.1109/access.2024.3449106 · Cited by 5 · see citing works
RetypeR: Integrated Retrieval-based Automatic Program Repair for Python Type Errors
Hao, S., Shi, X., & Liu, H. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00028 · Cited by 5 · see citing works
Towards understanding the challenges of bug localization in deep learning systems
Jahan, S., Shah, M. B., & Rahman, M. M. (2024) · Empirical Software Engineering · doi:10.1007/s10664-025-10707-0 · Cited by 7 · see citing works
Hierarchical Knowledge Injection for Improving LLM-based Program Repair
Ehsani, R., Parra, E., Haiduc, S., & Chatterjee, P. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00122 · Cited by 11 · see citing works
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 5 · see citing works
Predictive Program Slicing via Execution Knowledge-Guided Dynamic Dependence Learning
Yadavally, A., Li, Y., & Nguyen, T. N. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643739 · Cited by 5 · see citing works
Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review
Zhang, J., Hu, X., Gao, C., Xia, X., & Li, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802827 · Cited by 6 · see citing works
Looking for the Needle in the Haystack: End-to-end Tests in Open Source Projects
Gortázar, F., Maes-Bermejo, M., Gallego, M., & Padilla, J. C. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-85347-1_4 · Cited by 4 · see citing works
ExcePy: A Python Benchmark for Bugs with Python Built-in Types
Zhang, X., Yan, R., Yan, J., Cui, B., Yan, J., & Zhang, J. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00104 · Cited by 4 · see citing works
Fault Localization via Fine-Tuning Large Language Models with Mutation Generated Stack Traces
Jambigi, N., Bogacz, B., Mueller, M., Bach, T., & Felderer, M. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988982 · Cited by 4 · see citing works
Measuring the Influence of Incorrect Code on Test Generation
Huang, D., Zhang, J. M., Harman, M., Du, M., & Cui, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764532 · Cited by 7 · see citing works
SUCCEED: Sharing Upcycling Cases with Context and Evaluation for Efficient Software Development
Nakata, T., Chen, S., Saiki, S., & Nakamura, M. (2023) · Information · doi:10.3390/info14090518 · Cited by 4 · see citing works
Mining Bug Repositories for Multi-Fault Programs
Callaghan, D., & Fischer, B. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00023 · Cited by 4 · see citing works
Test Benchmarks: Which One Now and in Future?
Artho, C. V., Benali, A., & Ramler, R. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00044 · Cited by 4 · see citing works
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3 · see citing works
Self-citationTowards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection
Yan, Y., Duong, V., Shao, H., & Poshyvanyk, D. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00177 · Cited by 3 · see citing works
Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges
Nashid, N., Ding, D., Gallaba, K., Hassan, A. E., & Mesbah, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00137 · Cited by 3 · see citing works
P3: A Dataset of Partial Program Fixes
Beyer, D., Grunske, L., Kettl, M., Lingsch-Rosenfeld, M., & Raselimo, M. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644889 · Cited by 3 · see citing works
Naziri, M. M. A., Singh, A. K., Wu, B., Qin, F., Dutta, S., & d’Amorim, M. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731739 · Cited by 3 · see citing works
Investigating Reproducibility Challenges in LLM Bugfixing on the HumanEvalFix Benchmark
Szalontai, B., Márton, B., Pintér, B., & Gregorics, T. (2025) · Software · doi:10.3390/software4030017 · Cited by 3 · see citing works
Verification and Validation of Quantum Software
Fortunato, D., Jiménez-Navajas, L., Campos, J., & Abreu, R. (2024) · Springer eBooks · doi:10.1007/978-3-031-64136-7_5 · Cited by 3 · see citing works
BabelRTS: Polyglot Regression Test Selection
Maurina, G., Cazzola, W., & Ghosh, S. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3554403 · Cited by 2 · see citing works
Scalable Language Agnostic Taint Tracking using Explicit Data Dependencies
Effendi, S. D. B., Pinho, X., Dreyer, A. M., & Yamaguchi, F. (2025) · State Of the Art in Java Program Analysis · doi:10.1145/3735544.3735586 · Cited by 2 · see citing works
HaPy-Bug – Human Annotated Python Bug Resolution Dataset
Przymus, P., Fejzer, M., Narębski, J., Woźniak, R., Halada, Ł., Kazecki, A., Molchanov, M., & Stencel, K. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00024 · Cited by 2 · see citing works
Blended Analysis for Predictive Execution
Li, Y., Dhulipala, H., Yadavally, A., Rong, X., Wang, S., & Nguyen, T. N. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729402 · Cited by 2 · see citing works
PyResBugs: A Dataset of Residual Python Bugs for Natural Language-Driven Fault Injection
Cotroneo, D., Rosa, G. D., & Liguori, P. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00024 · Cited by 2 · see citing works
Large Language Models for Software Fault Localization: a Survey
Chang, X., Li, D., & Wong, W. E. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00024 · Cited by 2 · see citing works
Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs
Wang, J., Xie, X., Hu, Q., Liu, S., Yu, J., Kong, J., & Li, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00029 · Cited by 2 · see citing works
Unveiling the power of code pre-trained models in automated program repair: A systematic review
Zhan, S., Wang, X. J., Wei, D., Xiang, X., Lin, Y., Wu, H., & Lu, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113015 · Cited by 2 · see citing works
Kang, S., Yun, S., Hong, J., Yoo, S., & An, G. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786874 · Cited by 2 · see citing works
Zhang, J., Ristin, M., Schanely, P., van de Venn, H. W., & Gligoric, M. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558917 · Cited by 2 · see citing works
A Language-agnostic Framework for Mining Static Analysis Rules from Code Changes
Effendi, S. D. B., Çirişci, B., Mukherjee, R., Nguyen, H. A., & Tripp, O. (2023) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip58684.2023.00035 · Cited by 2 · see citing works
Yang, Y., Wang, Z., Chen, Z., & Xu, B. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111868 · Cited by 2 · see citing works
Adapting Knowledge Prompt Tuning for Enhanced Automated Program Repair
Cai, X., & Jiang, L. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00041 · Cited by 2 · see citing works
PatchScope – A Modular Tool for Annotating and Analyzing Contributions
Narębski, J., Fejzer, M., Stencel, K., & Przymus, P. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731727 · Cited by 2 · see citing works
Towards Understanding Fine-Grained Programming Mistakes and Fixing Patterns in Data Science
Chen, W., Cheoh, J. L., Keim, M., Brunswicker, S., & Zhang, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729352 · Cited by 2 · see citing works
- Authors
- Widyasari, R., Sim, S. Q., Lok, C., Qi, H., Phan, J., Tay, Q., Tan, C., Wee, F., Tan, J. E., Yieh, Y., Goh, B., Thung, F., Kang, H. J., Hoang, T., Lo, D., & Ouh, E. L.
- Affiliations
- Singapore Management University (SG)
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 1556-1560
- OpenAlex
- W3105943882
- Cited by
- 150
- References
- 17
- Citations by year
- 2026: 36 · 2025: 44 · 2024: 29 · 2023: 20 · 2022: 12 · 2021: 9
- FWCI
- 6.80
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Singapore Management University
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- Python, software defects, debugging tools, software testing, programming language popularity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
