The DOI matches its Crossref record; 17 of the 20 references it cites were found in a registry or an index, and 3 are not checked yet.
BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies
Open accessAbstract
The 2019 edition of Stack Overflow developer survey highlights that, for the first time, Python outperformed Java in terms of popularity. The gap between Python and Java further widened in the 2020 edition of the survey. Unfortunately, despite the rapid increase in Python's popularity, there are not many testing and debugging tools that are designed for Python. This is in stark contrast with the abundance of testing and debugging tools for Java. Thus, there is a need to push research on tools that can help Python developers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [35]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Matplotlib: a 2D graphics environment
Hunter, J. D. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.55 · Cited by 39,249
Reback, J., McKinney, W., Bossche, J. V. D., Jbrockmendel, Augspurger, T., Cloud, P., Gfyoung, Sinhrks, Klein, A., Tratner, J., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3509134 · Cited by 4,909
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Do, H., Elbaum, S. G., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,161
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 783
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635929 · Cited by 602
Honnibal, M., Montani, I., Honnibal, M., Peters, H., Samsonov, M., Geovedi, J., Regan, J., Orosz, G., Kristiansen, S. L., R., et al. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1212303 · Cited by 543
Le, X.-B. D., Lo, D., & Goues, C. L. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 341
Self-citationPrecise Condition Synthesis for Program Repair
Xiong, Y., Wang, J., Yan, R., Zhang, J., Shi, H., Huang, G., & Zhang, L. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.45 · Cited by 308
FLUCCS: using code and change metrics to improve fault localization
Sohn, J., & Yoo, S. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092717 · Cited by 252
Research works printed but not linked: 1
- Durieux Thomas, 1905, Critical Review of BugSwarm for Fault Localization and Program Repair. arXiv preprint arXiv
Cited by
Sorted by citations
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
Large Language Models for Test-Free Fault Localization
Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623342 · Cited by 144
A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
Kang, S., An, G., & Yoo, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660771 · Cited by 128
Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM
Ryan, G., Jain, S., Shang, M., Wang, S., Ma, X., Ramanathan, M. K., & Ray, B. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643769 · Cited by 180
An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Meng, X., Zhang, J., Liu, Y., Wang, W., Li, S., & Zhang, Y. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00181 · Cited by 113
A Survey on Software Defect Prediction Using Deep Learning
Akimova, E. N., Bersenev, A. Y., Deikov, A. A., Kobylkin, K. S., Konygin, A. V., Mezentsev, I. P., & Misilov, V. E. (2021) · Mathematics · doi:10.3390/math9111180 · Cited by 125
Explainable automated debugging via large language model-driven scientific debugging
Kang, S., Chen, B., Yoo, S., & Lou, J. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10594-x · Cited by 119
A Survey of Challenges in Spectrum-Based Software Fault Localization
Sarhan, Q. I., & Beszédes, Á. (2022) · IEEE Access · doi:10.1109/access.2022.3144079 · Cited by 78
An empirical study of automated unit test generation for Python
Lukasczyk, S., Kroiß, F., & Fraser, G. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10248-w · Cited by 66
Fixing dependency errors for Python build reproducibility
Mukherjee, S., Almanza, A., & Rubio-González, C. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464797 · Cited by 62
Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting
Li, T.-O., Zong, W., Wang, Y., Tian, H., Wang, Y., Cheung, S.-C., & Kramer, J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00089 · Cited by 70
CoverUp: Effective High Coverage Test Generation for Python
Pizzorno, J. A., & Berger, E. D. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729398 · Cited by 84
Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors
Peng, Y., Gao, S., Gao, C., Huo, Y., & Lyu, M. R.-T. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608132 · Cited by 66
Evaluating large language models for software testing
Li, Y., Liu, P., Wang, H., Chu, J., & Wong, W. E. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103942 · Cited by 42
Test Oracle Automation in the Era of LLMs
Molina, F., Gorla, A., & d’Amorim, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715107 · Cited by 54
You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects
Bouzenia, I., & Pradel, M. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3728922 · Cited by 35
Comprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair
Huang, K., Zhang, J., Bao, X., Wang, X., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3532759 · Cited by 30
Demystifying Faulty Code: Step-by-Step Reasoning for Explainable Fault Localization
Widyasari, R., Ang, J. W., Nguyen, T. G., Sharma, N., & Lo, D. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00064 · Cited by 36
Self-citationWidyasari, R., Prana, G. A. A., Haryono, S. A., Wang, S., & Lo, D. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10189-4 · Cited by 26
Self-citationFault localization to detect co-change fixing locations
Li, Y., Wang, S., & Nguyen, T. N. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549137 · Cited by 25
Bugs in machine learning-based systems: a faultload benchmark
Morovati, M. M., Nikanjam, A., Khomh, F., & Jiang, Z. M. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10291-1 · Cited by 31
PyTER: effective program repair for Python type errors
Oh, W.-S., & Oh, H. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549130 · Cited by 22
Out of Context: How important is Local Context in Neural Program Repair?
Prenner, J. A., & Robbes, R. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639086 · Cited by 26
The Fact Selection Problem in LLM-Based Program Repair
Parasaram, N., Yan, H., Yang, B., Flahy, Z., Qudsi, A., Ziaber, D., Barr, E. T., & Mechtaev, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00162 · Cited by 42
Mining Python fix patterns via analyzing fine-grained source code changes
Yang, Y., He, T., Feng, Y., Liu, S., & Xu, B. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10087-1 · Cited by 19
A systematic literature review on benchmarks for evaluating debugging approaches
Hirsch, T., & Hofer, B. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111423 · Cited by 20
Towards understanding bugs in Python interpreters
Liu, D., Feng, Y., Yan, Y., & Xu, B. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10239-x · Cited by 18
The GitHub Recent Bugs Dataset for Evaluating LLM-Based Debugging Applications
Lee, J. Y., Kang, S., Yoon, J., & Yoo, S. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00049 · Cited by 18
Are Large Language Models Memorizing Bug Benchmarks?
Ramos, D., Mamede, C., Jain, K., Canelas, P., Gamboa, C., & Goues, C. L. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00005 · Cited by 40
Optimizing Search-Based Unit Test Generation with Large Language Models: An Empirical Study
Xiao, D., Guo, Y., Li, Y., & Chen, L. (2024) · Asia-Pacific Symposium on Internetware · doi:10.1145/3671016.3674813 · Cited by 15
When Deep Learning Meets Information Retrieval-based Bug Localization: A Survey
Niu, F., Li, C., Liu, K., Xia, X., & Lo, D. (2025) · ACM Computing Surveys · doi:10.1145/3734217 · Cited by 21
Self-citationAkimova, E. N., Bersenev, A. Y., Deikov, A. A., Kobylkin, K. S., Konygin, A. V., Mezentsev, I. P., & Misilov, V. E. (2021) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec53868.2021.00022 · Cited by 15
Learning Realistic Mutations: Bug Creation for Neural Bug Detectors
Richter, C., & Wehrheim, H. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00027 · Cited by 19
Eberlein, M., Smytzek, M., Steinhöfel, D., Grunske, L., & Zeller, A. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616296 · Cited by 18
Tests4Py: A Benchmark for System Testing
Smytzek, M., Eberlein, M., Serçe, B., Grunske, L., & Zeller, A. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663798 · Cited by 13
Denchmark: A Bug Benchmark of Deep Learning-related Software
Kim, M., Kim, Y., & Lee, E. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00070 · Cited by 13
Zhang, J., Hu, X., Xia, X., Cheung, S.-C., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3745765 · Cited by 13
Fine-grained, accurate and scalable source differencing
Falleri, J., & Martínez, M. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639148 · Cited by 12
An empirical study of fault localization in Python programs
Rezaalipour, M., & Furia, C. A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10475-3 · Cited by 14
Rete: Learning Namespace Representation for Program Repair
Parasaram, N., Barr, E. T., & Mechtaev, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00112 · Cited by 12
BUGSC++: A Highly Usable Real World Defect Benchmark for C/C++
An, G., Kwon, M., Choi, K., Yi, J., & Yoo, S. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00208 · Cited by 12
DyPyBench: A Benchmark of Executable Python Software
Bouzenia, I., Krishan, B. P., & Pradel, M. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643742 · Cited by 11
T5APR: Empowering automated program repair across languages through checkpoint ensemble
Gharibi, R., Sadreddini, M. H., & Fakhrahmad, S. M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112083 · Cited by 15
Neural-MBFL: Improving Mutation-Based Fault Localization by Neural Mutation
Du, B., Han, B., Liu, H., Chang, Z., Liu, Y., & Chen, X. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00168 · Cited by 11
ChatDL: An LLM-Based Defect Localization Approach for Software in IIoT Flexible Manufacturing
Yang, H., Zhou, Y. L., Liang, T., & Kuang, L. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3531512 · Cited by 14
SFLKit: a workbench for statistical fault localization
Smytzek, M., & Zeller, A. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558915 · Cited by 11
VALIDATE: A deep dive into vulnerability prediction datasets
Esposito, M., & Falessi, D. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107448 · Cited by 13
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 40
A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing Techniques
Elgendy, I. T., Hierons, R. M., & McMinn, P. (2025) · Software Testing Verification and Reliability · doi:10.1002/stvr.1914 · Cited by 12
Reflective Unit Test Generation for Precise Type Error Detection with Large Language Models
Yang, C., Wang, Z., Jiang, Y., Yang, L., Zheng, Y., Zhou, J., & Chen, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00233 · Cited by 9
How Execution Features Relate to Failures: An Empirical Study and Diagnosis Approach
Smytzek, M., Eberlein, M., Grunske, L., & Zeller, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3783989 · Cited by 8
From Bugs to Benchmarks: A Comprehensive Survey of Software Defect Datasets
Zhu, H., Furth, R. M., Pradel, M., & Rubio-González, C. (2026) · ACM Computing Surveys · doi:10.1145/3797033 · Cited by 25
Demystifying regular expression bugs
Wang, P., Brown, C., Jennings, J. A., & Stolee, K. T. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10033-1 · Cited by 8
AWSomePy: A Dataset and Characterization of Serverless Applications
Raffa, G., Blasco, J., O’Keeffe, D., & Dash, S. K. (2023) · Workshop on SErverless Systems, Applications and MEthodologies (SESAME) · doi:10.1145/3592533.3592811 · Cited by 8
Integrating neural mutation into mutation-based fault localization: A hybrid approach
Liu, H., Li, Z., Han, B., Chen, X., Paul, D., & Liu, Y. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112281 · Cited by 8
Tanzil, M. H., Chowdhury, S., Modaberi, S., Uddin, G., & Hemmati, H. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112405 · Cited by 7
BugsPHP: A dataset for Automated Program Repair in PHP
Pramod, D., Silva, T. D., Thabrew, U., Shariffdeen, R., & Wickramanayake, S. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644878 · Cited by 7
A dataset of regressions in web applications detected by end-to-end tests
Soto-Sánchez, Ó., Maes-Bermejo, M., Gallego, M., & Gortázar, F. (2021) · Software Quality Journal · doi:10.1007/s11219-021-09566-x · Cited by 7
ROBUST: 221 bugs in the Robot Operating System
Timperley, C. S., van der Hoorn, G., SANTOS, A. F. D. M. S., Deshpande, H., & Wąsowski, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10440-0 · Cited by 7
RunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 33
Towards Effective Static Type-Error Detection for Python
Oh, W., & Oh, H. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695545 · Cited by 6
Yin, X., Ni, C., Li, X., Chen, L., Ma, G., & Yang, X. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00058 · Cited by 10
How Well Static Type Checkers Work with Gradual Typing? A Case Study on Python
Xu, W., Chen, L., Su, C., Guo, Y., Li, Y., Zhou, Y., & Xu, B. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00039 · Cited by 6
Ye, H., Chen, Z., & Goues, C. L. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00163 · Cited by 6
PyBugHive: A Comprehensive Database of Manually Validated, Reproducible Python Bugs
Antal, G., Vándor, N., Kolláth, I., Mosolygó, B., Hegedűs, P., & Ferenć, R. (2024) · IEEE Access · doi:10.1109/access.2024.3449106 · Cited by 5
RetypeR: Integrated Retrieval-based Automatic Program Repair for Python Type Errors
Hao, S., Shi, X., & Liu, H. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00028 · Cited by 5
Towards understanding the challenges of bug localization in deep learning systems
Jahan, S., Shah, M. B., & Rahman, M. M. (2024) · Empirical Software Engineering · doi:10.1007/s10664-025-10707-0 · Cited by 7
Hierarchical Knowledge Injection for Improving LLM-based Program Repair
Ehsani, R., Parra, E., Haiduc, S., & Chatterjee, P. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00122 · Cited by 11
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 13
Predictive Program Slicing via Execution Knowledge-Guided Dynamic Dependence Learning
Yadavally, A., Li, Y., & Nguyen, T. N. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643739 · Cited by 5
Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review
Zhang, J., Hu, X., Gao, C., Xia, X., & Li, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802827 · Cited by 6
Looking for the Needle in the Haystack: End-to-end Tests in Open Source Projects
Gortázar, F., Maes-Bermejo, M., Gallego, M., & Padilla, J. C. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-85347-1_4 · Cited by 4
ExcePy: A Python Benchmark for Bugs with Python Built-in Types
Zhang, X., Yan, R., Yan, J., Cui, B., Yan, J., & Zhang, J. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00104 · Cited by 4
Fault Localization via Fine-Tuning Large Language Models with Mutation Generated Stack Traces
Jambigi, N., Bogacz, B., Mueller, M., Bach, T., & Felderer, M. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988982 · Cited by 6
Measuring the Influence of Incorrect Code on Test Generation
Huang, D., Zhang, J. M., Harman, M., Du, M., & Cui, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764532 · Cited by 21
SUCCEED: Sharing Upcycling Cases with Context and Evaluation for Efficient Software Development
Nakata, T., Chen, S., Saiki, S., & Nakamura, M. (2023) · Information · doi:10.3390/info14090518 · Cited by 4
Mining Bug Repositories for Multi-Fault Programs
Callaghan, D., & Fischer, B. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00023 · Cited by 4
Test Benchmarks: Which One Now and in Future?
Artho, C. V., Benali, A., & Ramler, R. (2021) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs54544.2021.00044 · Cited by 4
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3
Self-citationTowards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection
Yan, Y., Duong, V., Shao, H., & Poshyvanyk, D. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00177 · Cited by 3
Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges
Nashid, N., Ding, D., Gallaba, K., Hassan, A. E., & Mesbah, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00137 · Cited by 3
P3: A Dataset of Partial Program Fixes
Beyer, D., Grunske, L., Kettl, M., Lingsch-Rosenfeld, M., & Raselimo, M. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644889 · Cited by 3
Naziri, M. M. A., Singh, A. K., Wu, B., Qin, F., Dutta, S., & d’Amorim, M. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731739 · Cited by 3
Investigating Reproducibility Challenges in LLM Bugfixing on the HumanEvalFix Benchmark
Szalontai, B., Márton, B., Pintér, B., & Gregorics, T. (2025) · Software · doi:10.3390/software4030017 · Cited by 3
Verification and Validation of Quantum Software
Fortunato, D., Jiménez-Navajas, L., Campos, J., & Abreu, R. (2024) · Springer eBooks · doi:10.1007/978-3-031-64136-7_5 · Cited by 3
BabelRTS: Polyglot Regression Test Selection
Maurina, G., Cazzola, W., & Ghosh, S. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3554403 · Cited by 2
Scalable Language Agnostic Taint Tracking using Explicit Data Dependencies
Effendi, S. D. B., Pinho, X., Dreyer, A. M., & Yamaguchi, F. (2025) · State Of the Art in Java Program Analysis · doi:10.1145/3735544.3735586 · Cited by 2
HaPy-Bug – Human Annotated Python Bug Resolution Dataset
Przymus, P., Fejzer, M., Narębski, J., Woźniak, R., Halada, Ł., Kazecki, A., Molchanov, M., & Stencel, K. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00024 · Cited by 2
Blended Analysis for Predictive Execution
Li, Y., Dhulipala, H., Yadavally, A., Rong, X., Wang, S., & Nguyen, T. N. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729402 · Cited by 2
PyResBugs: A Dataset of Residual Python Bugs for Natural Language-Driven Fault Injection
Cotroneo, D., Rosa, G. D., & Liguori, P. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00024 · Cited by 2
Large Language Models for Software Fault Localization: a Survey
Chang, X., Li, D., & Wong, W. E. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00024 · Cited by 2
Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs
Wang, J., Xie, X., Hu, Q., Liu, S., Yu, J., Kong, J., & Li, Y. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00029 · Cited by 2
Unveiling the power of code pre-trained models in automated program repair: A systematic review
Zhan, S., Wang, X. J., Wei, D., Xiang, X., Lin, Y., Wu, H., & Lu, J. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113015 · Cited by 2
Kang, S., Yun, S., Hong, J., Yoo, S., & An, G. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786874 · Cited by 2
Zhang, J., Ristin, M., Schanely, P., van de Venn, H. W., & Gligoric, M. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558917 · Cited by 2
A Language-agnostic Framework for Mining Static Analysis Rules from Code Changes
Effendi, S. D. B., Çirişci, B., Mukherjee, R., Nguyen, H. A., & Tripp, O. (2023) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip58684.2023.00035 · Cited by 2
Yang, Y., Wang, Z., Chen, Z., & Xu, B. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111868 · Cited by 2
Adapting Knowledge Prompt Tuning for Enhanced Automated Program Repair
Cai, X., & Jiang, L. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00041 · Cited by 3
PatchScope – A Modular Tool for Annotating and Analyzing Contributions
Narębski, J., Fejzer, M., Stencel, K., & Przymus, P. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731727 · Cited by 2
Towards Understanding Fine-Grained Programming Mistakes and Fixing Patterns in Data Science
Chen, W., Cheoh, J. L., Keim, M., Brunswicker, S., & Zhang, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729352 · Cited by 2
Metadata
- Authors
- Widyasari, R., Sim, S. Q., Lok, C., Qi, H., Phan, J., Tay, Q., Tan, C., Wee, F., Tan, J. E., Yieh, Y., Goh, B., Thung, F., Kang, H. J., Hoang, T., Lo, D., & Ouh, E. L.
- Affiliations
- Singapore Management University (SG)
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 1556-1560
- OpenAlex
- W3105943882
- Cited by
- 150
- References
- 17
- Citations by year
- 2026: 36 · 2025: 44 · 2024: 29 · 2023: 20 · 2022: 12 · 2021: 9
- FWCI
- 6.80
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Singapore Management University
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- Python, software defects, debugging tools, software testing, programming language popularity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
