Flakiness Repair in the Era of Large Language Models
Abstract
Flaky tests can non-deterministically pass or fail regardless of any change to the code, which negatively impacts the effectiveness of the regression testing. Prior repair techniques for flaky tests mainly leverage program analysis techniques to mitigate test flakiness, which only focus on Order-Dependent (OD) and Implementation-Dependent (ID) flakiness with known flakiness patterns and root causes. In this paper, we propose an approach to repair flaky tests with the power of Large Language Models (LLMs). Our approach successfully repaired 79% of OD tests and 58% of ID tests in an extensive evaluation using 666 flaky tests from 222 projects. We submitted pull requests to fix 61 flaky tests; at the time of submission, 19 tests have already been accepted. However, we observed that currently LLMs are ineffective in adequately repairing Non-Order-Dependent (NOD) flaky tests by analyzing 118 of such tests from 11 projects.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
International Conference on Software Engineering
(2017) · IEEE Software · doi:10.1109/ms.2017.3571566 · Cited by 622 · see citing works
Proceedings of the 44th International Conference on Software Engineering
(2022) · doi:10.1145/3510003 · Cited by 327 · see citing works
iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests
Lam, W., Oei, R., Shi, A., Marinov, D., & Xie, T. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00038 · Cited by 164 · see citing works
Root causing flaky tests in a large-scale industrial setting
Lam, W., Godefroid, P., Nath, S., Santhiar, A., & Thummalapenta, S. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330570 · Cited by 143 · see citing works
iFixFlakies: a framework for automatically fixing order-dependent flaky tests
Shi, A., Lam, W., Oei, R., Xie, T., & Marinov, D. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338925 · Cited by 119 · see citing works
A study on the lifecycle of flaky tests
Lam, W., Muşlu, K., Sajnani, H., & Thummalapenta, S. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3381749 · Cited by 115 · see citing works
What is the Vocabulary of Flaky Tests?
Pinto, G., Miranda, B., Dissanayake, S., d’Amorim, M., Treude, C., & Bertolino, A. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387482 · Cited by 101 · see citing works
Detecting flaky tests in probabilistic and machine learning applications
Dutta, S., Shi, A., Choudhary, R., Zhang, Z., Jain, A., & Misailovíc, S. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397366 · Cited by 79 · see citing works
Know You Neighbor: Fast Static Prediction of Test Flakiness
Verdecchia, R., Cruciani, E., Miranda, B., & Bertolino, A. (2021) · IEEE Access · doi:10.1109/access.2021.3082424 · Cited by 49 · see citing works
Identifying Key Success Factors in Stopping Flaky Tests in Automated REST Service Testing
Mascheroni, M. A., & Irrazábal, E. (2018) · Journal of Computer Science and Technology · doi:10.24215/16666038.18.e16 · Cited by 12 · see citing works
iPFlakies: A Framework for Detecting and Fixing Python Order-Dependent Flaky Tests
Wang, R., Chen, Y., & Lam, W. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion55297.2022.9793801 · Cited by 10 · see citing works
Self-citationFinding Polluter Tests Using Java PathFinder
Yi, P., Wei, A., Lam, W., Xie, T., & Marinov, D. (2021) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3468744.3468756 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Demystifying LLM-Based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715754 · Cited by 87 · see citing works
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77 · see citing works
Agentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24 · see citing works
“More recently, agent-based APR techniques have also been proposed [97, 29, 40, 32, 105].”
Neurosymbolic Repair of Test Flakiness
Chen, Y., & Jabbarvand, R. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680369 · Cited by 11 · see citing works
Self-citationDockerfile Flakiness: Characterization and Repair
Shabani, T., Nashid, N., Alian, P., & Mesbah, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00238 · Cited by 6 · see citing works
Can We Classify Flaky Tests Using Only Test Code? an LLM-Based Empirical Study
Berndt, A., Bekmyradov, V., Gemulla, R., Kessel, M., Bach, T., & Baltes, S. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00016
- Authors
- Chen, Y.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Corresponding author
- Chen, Yang
- ORCID
- Chen, Yang
- Venue
- Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings
- Publisher
- ACM
- Published
- 2024-04-14
- Pages
- 441-443
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4398239219
- Cited by
- 4
- References
- 14
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2
- FWCI
- 1.09
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- flaky tests, test repair, large language models, regression testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
