The DOI matches its Crossref record; 17 of the 20 references it cites were found in a registry or an index, and 3 are not checked yet.
BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies
Open accessAbstract
The 2019 edition of Stack Overflow developer survey highlights that, for the first time, Python outperformed Java in terms of popularity. The gap between Python and Java further widened in the 2020 edition of the survey. Unfortunately, despite the rapid increase in Python's popularity, there are not many testing and debugging tools that are designed for Python. This is in stark contrast with the abundance of testing and debugging tools for Java. Thus, there is a need to push research on tools that can help Python developers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We mitigate this by evaluating our approach on Defects4J [13] and BugsInPy [35], two widely used benchmarks containing diverse real-world faults in Java and Python projects.”
From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [35]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Matplotlib: a 2D graphics environment
Hunter, J. D. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.55 · Cited by 39,249
Reback, J., McKinney, W., Bossche, J. V. D., Jbrockmendel, Augspurger, T., Cloud, P., Gfyoung, Sinhrks, Klein, A., Tratner, J., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3509134 · Cited by 4,909
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Do, H., Elbaum, S. G., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,161
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 783
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635929 · Cited by 602
Honnibal, M., Montani, I., Honnibal, M., Peters, H., Samsonov, M., Geovedi, J., Regan, J., Orosz, G., Kristiansen, S. L., R., et al. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1212303 · Cited by 543
Le, X.-B. D., Lo, D., & Goues, C. L. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 341
Self-citationPrecise Condition Synthesis for Program Repair
Xiong, Y., Wang, J., Yan, R., Zhang, J., Shi, H., Huang, G., & Zhang, L. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.45 · Cited by 308
FLUCCS: using code and change metrics to improve fault localization
Sohn, J., & Yoo, S. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092717 · Cited by 252
Research works printed but not linked: 1
- Durieux Thomas, 1905, Critical Review of BugSwarm for Fault Localization and Program Repair. arXiv preprint arXiv
Cited by
Sorted by citations
Items 101–152 of 152
Towards Context-Aware Spectrum-Based Fault Localization
Szatmári, A. (2023) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst57152.2023.00060 · Cited by 2
Commit-Based Class-Level Defect Prediction for Python Projects
Khine Yin, M. O. N., Kondo, M., Choi, E., & Mizuno, O. (2023) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2022mpp0003 · Cited by 1
MultiMend: multilingual program repair with context augmentation and multi-hunk patch generation
Gharibi, R., Sadreddini, M. H., & Fakhrahmad, S. M. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00611-2 · Cited by 7
PAFL: Enhancing Fault Localizers by Leveraging Project-Specific Fault Patterns
Kim, D., Jeon, M., Hwang, D., & Oh, H. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3720526 · Cited by 1
Yang, Y., Li, C., Han, Z., Li, R., Xu, K., Li, Q., Zhong, W., Shen, Z., Fei, Z., Ge, J., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3764584 · Cited by 7
Fu, A., Xu, P., Li, J., Kuang, B., & Gao, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3618407 · Cited by 4
van der Spuy, A., & Fischer, B. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3757025 · Cited by 1
Lee, H., & Yang, G. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00025 · Cited by 1
Recovering Revisions of Pull Requests with Altered History
Anonymous, Randrianaina, G. A., Xiao, T., Kamei, Y., & McIntosh, S. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837417 · Cited by 1
HAFix: history-augmented large language models for bug fixing
Shi, Y., Bangash, A. A., Fallahzadeh, E., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10833-3 · Cited by 8
An Automated Methodology for Generating Labeled Datasets of Semantic Errors in Code
Kassem, M., Ribeiro, F., & Nadi, S. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788730 · Cited by 1
Pruning Boolean Expressions to Shorten Dynamic Slices
Hirsch, T., & Hofer, B. (2022) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam55253.2022.00006 · Cited by 1
Reducing the Length of Dynamic and Relevant Slices by Pruning Boolean Expressions
Hirsch, T., & Hofer, B. (2024) · Electronics · doi:10.3390/electronics13061146 · Cited by 1
Evaluating Source Code Comment Quality: The Need for Comprehensive Indexes in Software Development
Adak, M. F., & Kilic, M. A. E. (2024) · International Conference on Artificial Intelligence, Metaverse and Cybersecurity (ICAMAC) · doi:10.1109/icamac62387.2024.10829019 · Cited by 1
Lachesis: Predicting LLM Inference Accuracy using Structural Properties of Reasoning Paths
Kim, N., Kang, S., An, G., & Yoo, S. (2025) · IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest) · doi:10.1109/deeptest66595.2025.00008 · Cited by 4
Which Inputs Trigger My Patch?
Eberlein, M., Raselimo, M., & Grunske, L. (2025) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr66717.2025.00006 · Cited by 1
FLITSR: Improved Spectrum-Based Localization of Multiple Faults by Iterative Test Suite Reduction
Callaghan, D., & Fischer, B. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3745027 · Cited by 1
Rafi, M. N., Kim, D. J., Chen, T.-H., & Wang, S. (2024) · arXiv · doi:10.48550/arxiv.2412.18750
“0.0) [13] and 207 faults from 11 projects from the BugsInPy benchmark [35].”
InfluentialSERA: Soft-Verified Efficient Repository Agents
Shen, E., Tormoen, D., Shah, S., Farhadi, A., & Dettmers, T. (2026) · arXiv · doi:10.48550/arxiv.2601.20789
Wang, P., Brown, C., Jennings, J. A., & Stolee, K. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09693
From Input to Failure: Explaining Program Behavior via Cause-Effect Chains
Smytzek, M. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00081
Reproducing and Improving the BugsInPy Dataset
Aguilar, F., Grayson, S., & Marinov, D. (2023) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam59687.2023.00036
Deep Learning for Software Defect Prediction: An LSTM-based Approach
Sahatiya, E. A. P. (2023) · International Journal on Recent and Innovation Trends in Computing and Communication · doi:10.17762/ijritcc.v11i9.9920
Methods for Enhancing Software Fault Localization
Sarhan, Q. I. (2023) · doi:10.14232/phd.11772
A Survey on Code Pre-Trained Models for Neural Program RepairPreprint
Zhan, S., Wang, X., Wei, D., & Cao, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4831589
Towards understanding the security issues of Python programs
Fan, H., Liu, D., Wu, J., Feng, Y., Shi, Q., & Xu, B. (2025) · Asia-Pacific Symposium on Internetware · doi:10.1145/3755881.3755886
BASHIRI: Learning Failure Oracles from Execution Features
Smytzek, M., Eberlein, M., Mammadov, T., Grunske, L., & Zeller, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00372
Meta-agentic Framework for Software bug detection using Large Language Models
Padmanabhuni, S., Karuturi, B., Indupalli, J. K., Chilla, S. R., & Yelleti, V. (2026) · International Conference on Communication Systems & Networks (COMSNETS) · doi:10.1109/comsnets67989.2026.11418178 · Cited by 1
Integrating AI and knowledge systems to automate bug reporting in software testing
Adel, R., Youness, F., Kolkila, A. A. E.-A., Al-Marakeby, A., & Nagdy, N. M. (2026) · Discover Applied Sciences · doi:10.1007/s42452-026-08624-y
CDBench: Benchmarking the mutation testing capabilities of LLMs with code defenders
Romazanov, A., Fraser, G., & Herbold, S. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10901-8
Understanding Bug-Reproducing Tests: A First Empirical Study
Hora, A., & Fraser, G. (2026) · doi:10.1145/3793654.3793752
Progressively Mitigating API Hallucination in LLM-Generated Code via Knowledge Graph Reasoning
Li, Y., Ma, Z., Zou, Y., Wang, Y., Yang, L., & Xie, B. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00086
Will It Break in Production? Metric-Driven Prediction of Residual Defects in Python Systems
Rosa, G. D., & Liguori, P. (2026) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn69566.2026.00076
Understanding and Overcoming Limitations of Automated Test Generation
Krodinger, L. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00012
Round-Trip Mutation Testing: Translating Code to Natural Language Intent and back
Hamidi, A., Richter, C., Khanfir, A., & Papadakis, M. (2026) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw72326.2026.00050
Who Wrote This Patch? Toward Accountable Automated Program Repair
Ran, H.-J., Zhang, H., Kim, K., & Tang, X. (2026) · SIGSOFT FSE Companion · doi:10.1145/3803437.3805552
Chen, F., Yang, G., Lee, C., Jie, Z., & Chen, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3831589
Type-aware LLM-based Test Generation for Python Programs
Liu, R., Zhang, Z., Hu, Y., Lin, Y., Gao, X., & Sun, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3830083 · Cited by 10
Evaluating Agent Frameworks for Code-centric Software Engineering: A Comprehensive Empirical StudyPreprint
Yin, Z., Gao, C., Fan, C., Yang, W., & Xue, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7202151
Bui, T., Zhang, T., Thung, F., Xiong, Y., Jiang, P., Zhou, X., & Lo, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-5737-9_5 · Cited by 2
Self-citationTowards Quantum Program Bug Datasets and Benchmarking
Li, T., Yu, J., Fan, Z., Wang, J., Zhai, Y., Zhang, S., Peng, Y., Chen, Y., Chen, J., Kong, F., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817560
ReviewTestFuse: Fusing review-guided and direct test generation for deployable regression testing
Zhen, H., & Zhang, Z. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00669-y
An, D., Wang, S., Liu, B., & Zhu, L. (2026) · Mathematics · doi:10.3390/math14183297
Synthetic Repo-level Bug Dataset for Training Automated Program Repair Models
Minh, P. V. T., Phan, H. N., Phan, N. H., Le, C. A., Nguyen, T. N., & Bui, N. D. Q. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764562
Bug Whisperer: Unleashing LLMs to Craft Context-Aware Tests Guided by Path Constraints and Bug Clues
Xia, X., Zhang, J., Hu, X., & Li, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3846194
Jonnakuti, S. R. (2026) · International Conference on Modern Sustainable Systems (CMSS) · doi:10.1109/cmss69636.2026.11688270
MockWeaver: Mock-Aware Multi-Agent Unit Test Generation for External DependenciesPreprint
Dai, L., Zhang, T., Wang, Q., Liu, X., Jing, Y., Duan, H., & Shi, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7506899
Accurate Specification Recovery for Effective Test Generation
Zhao, J. (2026) · ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH) · doi:10.1145/3837729.3842146
Uncovering Business Logic Bugs via Semantics-Driven Unit Test Generation (Experience Paper)
Yang, C., & Chen, J.-J. (2026) · Proceedings of the ACM on Software Engineering · doi:10.1145/3832192 · Cited by 2
Steering Tree-of-Thought Reasoning via Deductive Verification
Haoliang, C., Tang, E., Zhang, S., Mao, J., Fu, Y., Ma, J., Cui, K., Liu, Y., Tian, Y., Gao, X., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832200
CodeFault: Predicting Fault Risks from Code Changes via Multi-modal Learning
XIAO, Y., Li, S., & Huang, Y. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837462
LLM-Pyro: LLM-Based Mutation Testing with Equivalence Triage for Python
Chang, V. M., Filho, M. M. D. L., & Fontão, A. (2026) · Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (SAST) · doi:10.5753/sast.2026.31353
Metadata
- Authors
- Widyasari, R., Sim, S. Q., Lok, C., Qi, H., Phan, J., Tay, Q., Tan, C., Wee, F., Tan, J. E., Yieh, Y., Goh, B., Thung, F., Kang, H. J., Hoang, T., Lo, D., & Ouh, E. L.
- Affiliations
- Singapore Management University (SG)
- Venue
- Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2020-11-08
- Pages
- 1556-1560
- OpenAlex
- W3105943882
- Cited by
- 150
- References
- 17
- Citations by year
- 2026: 36 · 2025: 44 · 2024: 29 · 2023: 20 · 2022: 12 · 2021: 9
- FWCI
- 6.80
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Singapore Management University
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- Python, software defects, debugging tools, software testing, programming language popularity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
