The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
A large-scale longitudinal study of flaky tests
Open accessAbstract
Flaky tests are tests that can non-deterministically pass or fail for the same code version. These tests undermine regression testing efficiency, because developers cannot easily identify whether a test fails due to their recent changes or due to flakiness. Ideally, one would detect flaky tests right when flakiness is introduced, so that developers can then immediately remove the flakiness. Some software organizations, e.g., Mozilla and Netflix, run some tools—detectors—to detect flaky tests as soon as possible. However, detecting flaky tests is costly due to their inherent non-determinism, so even state-of-the-art detectors are often impractical to be used on all tests for each project change. To combat the high cost of applying detectors, these organizations typically run a detector solely on newly added or directly modified tests, i.e., not on unmodified tests or when other changes occur (including changes to the test suite, the code under test, and library dependencies). However, it is unclear how many flaky tests can be detected or missed by applying detectors in only these limited circumstances. To better understand this problem, we conduct a large-scale longitudinal study of flaky tests to determine when flaky tests become flaky and what changes cause them to become flaky. We apply two state-of-theart detectors to 55 Java projects, identifying a total of 245 flaky tests that can be compiled and run in the code version where each test was added. We find that 75% of flaky tests (184 out of 245) are flaky when added, indicating substantial potential value for developers to run detectors specifically on newly added tests. However, running detectors solely on newly added tests would still miss detecting 25% of flaky tests. The percentage of flaky tests that can be detected does increase to 85% when detectors are run on newly added or directly modified tests. The remaining 15% of flaky tests become flaky due to other changes and can be detected only when detectors are always applied to all tests. Our study is the first to empirically evaluate when tests become flaky and to recommend guidelines for applying detectors in the future.
Verifying this paper
8 of 9 steps done · this page updates itself — you can leave and come back
- Registry recordVerified
- Full textRead the full text
- Reference list37 references found
- Reference checkChecking each cited work — 44 of 44
- StatisticsNeeds the full text
- ClaimsNeeds the full text
- Cited sourcesNeeds claims to check
- Self-checkNeeds the full text
- VerificationFindings verified
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
If the model generates a flaky test suite, it is possible that all tests pass on an incorrect solution and lead to a false positive label on a code completion.
“Therefore, in the case in which the model generates a flaky test suite, it is possible that all tests pass on an incorrect solution and lead to a false positive label on a code completion [11].”
From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [11]
The Introduction (page not provided) says a flaky test can pass and fail on the same code and that changes may be merged when all tests pass. It does not address model-generated test suites, incorrect code completions, or false-positive labels, so those parts of the claim are not established by this paper. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460
Regression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing Verification and Reliability · doi:10.1002/stv.430 · Cited by 1,330
Influence functions of the Spearman and Kendall correlation measures
Croux, C., & Dehon, C. (2010) · Statistical Methods & Applications · doi:10.1007/s10260-010-0142-z · Cited by 634 · cited 1×
“For the correlation analysis of flaky test numbers and project characteristics, we chose Kendall’s 𝜏 , as it is non-parametric and more robust than Spearman’s 𝜌 [Croux and Dehon 2010].”
MethodAn empirical analysis of flaky tests
Luo, Q., Hariri, F., Eloussi, L., & Marinov, D. (2014) · SIGSOFT FSE · doi:10.1145/2635868.2635920 · Cited by 551 · cited 4×
“Our literature search reveals only two research papers [Gambi et al. 2018; Luo et al. 2014] that mention checking whether tests are flaky when they are added, but both studies are quite limited to just a handful of flaky tests that were manually studied, and in at least one study [Luo et al. 2014]…”
BackgroundInfluentialSelf-citationYesterday, my program worked. Today, it does not. Why?
Zeller, A. (1999) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/318774.318946 · Cited by 371 · cited 1×
“In theory, we could use delta debugging [Zeller 1999] to automatically identify the change (or, more precisely, a set of changes), but in practice, there are no readily available tools for Java and Maven projects.”
MethodTaming Google-scale continuous testing
Memon, A. M., Gao, Z., Nguyen, B., Dhanda, S., Nickell, E., Siemborski, R., & Micco, J. (2017) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2017.16 · Cited by 245 · cited 1×
“Much existing work [Coverity 2014; Infer 2020; Larus et al. 2004; Memon et al. 2017; Muşlu et al. 2015; Sadowski et al. 2015; Saff and Ernst 2003] has found that providing developers with feedback regarding problematic code is most helpful if provided as soon as possible in the development process.”
Program Development in Java: Abstraction, Specification, and Object-Oriented Design
Liskov, B. H., & Guttag, J. V. (1986) · International Linear Collider · Cited by 222
A randomized scheduler with probabilistic guarantees of finding bugs
Burckhardt, S., Kothari, P. K., Musuvathi, M., & Nagarakatte, S. (2010) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/1736020.1736040 · Cited by 215 · cited 2×
“Running such detectors is generally costly (i.e., they require multiple runs of the entire test suite or at least some tests, using various random seeds or other causes of “noise” [Burckhardt et al. 2010]), so projects typically do not run these detectors on all tests for each and every commit.”
BackgroundMethodUnderstanding flaky tests: the developer’s perspective
Eck, M., Palomba, F., Castelluccio, M., & Bacchelli, A. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338945 · Cited by 173 · cited 3×
“Other organizations, including Mozilla [Eck et al. 2019; Rahman and Rigby 2018], Netflix [NetflixAutomationTalk 2017], Salesforce [Salesforce-FlakyTests 2016], and ThoughtWorks [Sudarshan 2012], also publicly report their problems with flaky tests.”
BackgroundBell, J., Legunsen, O., Hilton, M. L., Eloussi, L., Yung, T., & Marinov, D. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180164 · Cited by 170 · cited 1×
“Several projects have proposed techniques for detecting flaky tests [Bell et al. 2018; Gambi et al. 2018; Pinto et al. 2020; Terragni et al. 2020; Zhang et al. 2014].”
MethodSelf-citationiDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests
Lam, W., Oei, R., Shi, A., Marinov, D., & Xie, T. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00038 · Cited by 164 · cited 14×
“…and blog posts from Google [AvoidingFlakeyTests 2019; FlakinessDashboardHOWTO 2020; Micco 2020; Ziftci and Reardon 2017] and Microsoft [Harry 2019; Lam et al. 2019a, 2020a] have reported several challenges with flaky tests, even estimating the monetary cost of flaky tests on developer…”
BackgroundMethodResultInfluentialSelf-citationEmpirically revisiting the test independence assumption
Zhang, S., Jalali, D., Wuttke, J., Muşlu, K., Lam, W., Ernst, M. D., & Notkin, D. S. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610404 · Cited by 159 · cited 7×
“To help researchers and developers on this topic, this paper answers the following research questions: [Gambi et al. 2018; Gyori et al. 2015; Huo and Clause 2014; Lam et al. 2019b; Shi et al. 2016; Zhang et al. 2014].”
BackgroundMethodInfluentialSelf-citationYesterday, my Program Worked. Today, it Does Not. Why?
Zeller, A. (1999) · Lecture notes in computer science · doi:10.1007/3-540-48166-4_16 · Cited by 156
From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis
Harman, M., & O’Hearn, P. W. (2018) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2018.00009 · Cited by 155 · cited 2×
“For example, for the static bug-detection tool Infer [2020] at Facebook, Harman and O’Hearn [2018] found that bugs reported to developers at “post land” (after code has been merged into the SUT) had a close to 0% fix rate, yet bugs reported at “diff time” (when the developer submits the code change) had a fix rate of…”
MethodTricorder: Building a Program Analysis Ecosystem
Sadowski, C., van Gogh, J., Jaspan, C. N. C., Söderberg, E., & Winter, C. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.76 · Cited by 152
Reducing wasted development time via continuous testing
Saff, D., & Ernst, M. D. (2005) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2003.1251050 · Cited by 146 · cited 1×
“Much existing work [Coverity 2014; Infer 2020; Larus et al. 2004; Memon et al. 2017; Muşlu et al. 2015; Sadowski et al. 2015; Saff and Ernst 2003] has found that providing developers with feedback regarding problematic code is most helpful if provided as soon as possible in the development process.”
Root causing flaky tests in a large-scale industrial setting
Lam, W., Godefroid, P., Nath, S., Santhiar, A., & Thummalapenta, S. (2019) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3293882.3330570 · Cited by 143 · cited 14×
“…and blog posts from Google [AvoidingFlakeyTests 2019; FlakinessDashboardHOWTO 2020; Micco 2020; Ziftci and Reardon 2017] and Microsoft [Harry 2019; Lam et al. 2019a, 2020a] have reported several challenges with flaky tests, even estimating the monetary cost of flaky tests on developer…”
BackgroundMethodResultInfluentialSelf-citationiFixFlakies: a framework for automatically fixing order-dependent flaky tests
Shi, A., Lam, W., Oei, R., Xie, T., & Marinov, D. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338925 · Cited by 119 · cited 9×
“) The test does not fail when run in the test suite, because the test suite contains a “state-setter” [Shi et al. 2019] that happens to run before the brittle, but the test does fail when run by itself.”
BackgroundMethodResultInfluentialSelf-citationA study on the lifecycle of flaky tests
Lam, W., Muşlu, K., Sajnani, H., & Thummalapenta, S. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3381749 · Cited by 115 · cited 3×
“…and blog posts from Google [AvoidingFlakeyTests 2019; FlakinessDashboardHOWTO 2020; Micco 2020; Ziftci and Reardon 2017] and Microsoft [Harry 2019; Lam et al. 2019a, 2020a] have reported several challenges with flaky tests, even estimating the monetary cost of flaky tests on developer…”
BackgroundSelf-citationWhat is the Vocabulary of Flaky Tests?
Pinto, G., Miranda, B., Dissanayake, S., d’Amorim, M., Treude, C., & Bertolino, A. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387482 · Cited by 101 · cited 2×
“Pinto et al. [2020] proposed using machine learning techniques based on tokens obtained from test-method code to detect flaky tests.”
MethodLarus, J. R., Ball, T., Das, M., DeLine, R. A., Fähndrich, M. A., Pincus, J., Rajamani, S. K., & Venkatapathy, R. (2004) · IEEE Software · doi:10.1109/ms.2004.1293079 · Cited by 97 · cited 1×
“Much existing work [Coverity 2014; Infer 2020; Larus et al. 2004; Memon et al. 2017; Muşlu et al. 2015; Sadowski et al. 2015; Saff and Ernst 2003] has found that providing developers with feedback regarding problematic code is most helpful if provided as soon as possible in the development process.”
Reliable testing: detecting state-polluting tests to prevent test dependency
Gyori, A., Shi, A., Hariri, F., & Marinov, D. (2015) · International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771793 · Cited by 91 · cited 5×
“To help researchers and developers on this topic, this paper answers the following research questions: [Gambi et al. 2018; Gyori et al. 2015; Huo and Clause 2014; Lam et al. 2019b; Shi et al. 2016; Zhang et al. 2014].”
BackgroundMethodInfluentialSelf-citationThe Art of Testing Less without Sacrificing Quality
Herzig, K., Greiler, M., Czerwonka, J., & Murphy, T. B. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.66 · Cited by 88 · cited 1×
“2019a, 2020a] have reported several challenges with flaky tests, even estimating the monetary cost of flaky tests on developer productivity [Herzig et al. 2015].”
BackgroundPractical Test Dependency Detection
Gambi, A., Bell, J., & Zeller, A. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00011 · Cited by 85 · cited 4×
“Aside from rerunning tests in isolation, various other approaches have been proposed to detect flaky tests [Gambi et al. 2018; Gyori et al. 2015; Huo and Clause 2014; Lam et al. 2019b; Shi et al. 2016; Zhang et al. 2014].”
BackgroundMethodInfluentialSelf-citationUnit test virtualization with VMVM
Bell, J., & Kaiser, G. E. (2014) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/2568225.2568248 · Cited by 84 · cited 1×
“Another line of prior work [Bell 2014; Bell and Kaiser 2014; Muşlu et al. 2011] has proposed isolating tests from each other during their runs.”
ResultSelf-citationJiang, H., Li, X., Yang, Z., & Xuan, J. (2017) · International Conference on Software Engineering · doi:10.1109/icse.2017.71 · Cited by 70
Lam, W., Winter, S., Astorga, A., Stodden, V., & Marinov, D. (2020) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre5003.2020.00045 · Cited by 66 · cited 3×
“…and blog posts from Google [AvoidingFlakeyTests 2019; FlakinessDashboardHOWTO 2020; Micco 2020; Ziftci and Reardon 2017] and Microsoft [Harry 2019; Lam et al. 2019a, 2020a] have reported several challenges with flaky tests, even estimating the monetary cost of flaky tests on developer…”
BackgroundSelf-citationThe art of testing less without sacrificing quality
Herzig, K., Greiler, M., Czerwonka, J., & Murphy, T. B. (2015) · doi:10.5555/2818754.2818815 · Cited by 55
Improving oracle quality by detecting brittle assertions and unused inputs in tests
Huo, C., & Clause, J. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635917 · Cited by 55
Ziftci, C., & Reardon, J. (2017) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2017.13 · Cited by 55 · cited 1×
“Several papers and blog posts from Google [AvoidingFlakeyTests 2019; FlakinessDashboardHOWTO 2020; Micco 2020; Ziftci and Reardon 2017] and Microsoft [Harry 2019; Lam et al. 2019a, 2020a] have reported several challenges with flaky tests, even estimating the monetary cost of flaky tests on…”
BackgroundModeling and ranking flaky tests at Apple
Kowalczyk, E., Nair, K., Gao, Z., Silberstein, L., Long, T., & Memon, A. M. (2020) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3377813.3381370 · Cited by 54 · cited 1×
“In recent years we have seen a number of studies on the characteristics of flaky tests [Eck et al. 2019; Gao et al. 2015; Jiang et al. 2017; Kowalczyk et al. 2020; Lam et al. 2020a, 2019b, 2020b; Luo et al. 2014].”
BackgroundFinding bugs by isolating unit tests
Muşlu, K., Soran, B., & Wuttke, J. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025202 · Cited by 41 · cited 1×
“Another line of prior work [Bell 2014; Bell and Kaiser 2014; Muşlu et al. 2011] has proposed isolating tests from each other during their runs.”
ResultPreventing data errors with continuous testing
Muşlu, K., Brun, Y., & Meliou, A. (2015) · International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771792 · Cited by 34 · cited 1×
“Much existing work [Coverity 2014; Infer 2020; Larus et al. 2004; Memon et al. 2017; Muşlu et al. 2015; Sadowski et al. 2015; Saff and Ernst 2003] has found that providing developers with feedback regarding problematic code is most helpful if provided as soon as possible in the development process.”
Influence Functions of the Spearman and Kendall Correlation MeasuresPreprint
Croux, C., & Dehon, C. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1585216 · Cited by 32
Making System User Interactive Tests Repeatable: When and What Should We Control?
Gao, Z., Liang, Y., Cohen, M. B., Memon, A. M., & Wang, Z. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.28 · Cited by 32
A container-based infrastructure for fuzzy-driven root causing of flaky tests
Terragni, V., Salza, P., & Ferrucci, F. (2020) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3377816.3381742 · Cited by 30 · cited 1×
“Several projects have proposed techniques for detecting flaky tests [Bell et al. 2018; Gambi et al. 2018; Pinto et al. 2020; Terragni et al. 2020; Zhang et al. 2014].”
MethodDetecting, isolating, and enforcing dependencies among and within test cases
Bell, J. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2666597 · Cited by 13 · cited 1×
“Another line of prior work [Bell 2014; Bell and Kaiser 2014; Muşlu et al. 2011] has proposed isolating tests from each other during their runs.”
ResultSelf-citationVMVM: unit test virtualization for Java
Bell, J., & Kaiser, G. (2014) · Companion Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2591062.2591063
Self-citationA randomized scheduler with probabilistic guarantees of finding bugs
Burckhardt, S., Kothari, P., Musuvathi, M., & Nagarakatte, S. (2010) · ACM SIGPLAN Notices · doi:10.1145/1735971.1736040
Making System User Interactive Tests Repeatable: When and What Should we Control?
Gao, Z. (2016) · 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2016.53
Debugging the performance of Maven’s test isolation: experience report
Nie, P., Celik, A., Coley, M., Milicevic, A., Bell, J., & Gligoric, M. (2020) · Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3395363.3397381
Self-citationRahman, M. T., & Rigby, P. C. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3275529
Detecting Assumptions on Deterministic Implementations of Non-deterministic Specifications
Shi, A., Gyori, A., Legunsen, O., & Marinov, D. (2016) · 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2016.40
Self-citationRegression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stvr.430
Flagged first, then by citations
Research works printed but not linked: 1
- Jonathan Bell Owolabi Legunsen Michael Hilton Lamyaa Eloussi Tifany Yung and Darko Marinov. 2018. DeFlaker: Automatically detecting flaky tests. In ICSE. Jonathan Bell Owolabi Legunsen Michael Hilton Lamyaa Eloussi Tifany Yung and Darko Marinov. 2018. DeFlaker: Automatically detecting flaky tests. In ICSE.
Cited by
Sorted by citations
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
“Therefore, in the case in which the model generates a flaky test suite, it is possible that all tests pass on an incorrect solution and lead to a false positive label on a code completion [11].”
Parry, O., Kapfhammer, G. M., Hilton, M. L., & McMinn, P. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3476105 · Cited by 189
Free lunch for testing: fuzzing deep-learning libraries from open source
Wei, A., Deng, Y., Yang, C., & Zhang, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510041 · Cited by 106
“This type of tests are called implementation-dependent flaky tests, described in prior work on test flakiness [41, 50, 72].”
MethodSelf-citationFlakeFlagger: Predicting Flakiness Without Rerunning Tests
Alshammari, A., Morris, C., Hilton, M. L., & Bell, J. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00140 · Cited by 75
“For instance, when a new test is introduced, developers might use FlakeFlagger to identify if that new test is likely to become flaky — since research has shown that most flaky tests are flaky when they are introduced [27].”
BackgroundSelf-citationFlakify: A Black-Box, Language Model-Based Predictor for Flaky Tests
Fatima, S., Ghaleb, T. A., & Briand, L. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3201209 · Cited by 78
“flaky test cases used by previous studies on flaky test case prediction [5], [50], [51], [52], [53], [54].”
MethodCoverUp: Effective High Coverage Test Generation for Python
Pizzorno, J. A., & Berger, E. D. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729398 · Cited by 84
“Flaky tests are those that can both pass or fail inconsistently without any changes to the code under test [Gruber et al. 2024; Lam et al. 2020; Parry et al. 2021].”
Know You Neighbor: Fast Static Prediction of Test Flakiness
Verdecchia, R., Cruciani, E., Miranda, B., & Bertolino, A. (2021) · IEEE Access · doi:10.1109/access.2021.3082424 · Cited by 49
“[31] investigates at which commit flakiness is introduced, and finds that 85% of flaky tests are already so when added or modified, but the remaining 15% ones become flaky later at unrelated commits.”
BackgroundContinuous Integration and Delivery Practices for Cyber-Physical Systems: An Interview-Based Study
Zampetti, F., Tamburri, D. A., Panichella, S., Panichella, A., Canfora, G., & Penta, M. D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3571854 · Cited by 42
“Flakiness related to non-determinism during test execution [89] has been largely studied [16, 46, 48, 58, 90] and approaches to detect and cope with it have been proposed [47, 49, 59, 65, 87].”
BackgroundA Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests
Haben, G., Habchi, S., Papadakis, M., Cordy, M., & Traon, Y. L. (2021) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00034 · Cited by 34
“[17] also carried a large-scale study on flaky tests, attempting to understand when tests first become flaky.”
BackgroundDomain-Specific Fixes for Flaky Tests with Wrong Assumptions on Underdetermined Specifications
Zhang, P., Jiang, Y., Wei, A., Stodden, V., Marinov, D., & Shi, A. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00018 · Cited by 29
“[48] recently used NonDex in a larger study of flaky tests, which can fail seemingly nondeterministically [53].”
BackgroundMethodSelf-citationAn Empirical Study of Flaky Tests in JavaScript
Hashemi, N., Tahir, A., & Rasheed, S. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00011 · Cited by 28
“[23] conducted a large-scale longitudinal study of flaky tests to determine when those tests become flaky, and what changes cause them to become so.”
BackgroundiPFlakies: a framework for detecting and fixing python order-dependent flaky tests
Wang, R., Chen, Y., & Lam, W. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3516846 · Cited by 36
“Since the taxonomy, much of the flaky-test work has focused on OD tests [1, 2, 7–9, 12, 15, 19].”
BackgroundSelf-citationTest flakiness’ causes, detection, impact and responses: A multivocal review
Tahir, A., Rasheed, S., Dietrich, J. B., Hashemi, N., & Zhang, L. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111837 · Cited by 24
Preempting flaky tests via non-idempotent-outcome tests
Wei, A., Yi, P., Li, Z., Xie, T., Marinov, D., & Lam, W. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510170 · Cited by 24
“To understand how the NIO tests that we detect relate to OD-related tests detected in prior work, we intentionally use the same projects/modules and commits as prior studies for Java [61, 92] and Python [39] that have reported some victims or polluters.”
BackgroundMethodInfluentialSelf-citationFlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair
Fatima, S., Hemmati, H., & Briand, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3472476 · Cited by 35
“Several other companies like Microsoft [3], Facebook [4], Netflix, Mozilla, and Sales-force [5] have reported similar experiences, where flaky tests have significantly delayed their deployment process.”
BackgroundMethodRepairing order-dependent flaky tests via test generation
Li, C., Zhu, C., Wang, W., & Shi, A. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510173 · Cited by 22
“Luo et al. found that order-dependent tests are one of the most common kinds of flaky tests in their empirical study [37], and much work has followed that focuses on such flaky tests [27, 28, 32–35, 43– 45, 48].”
BackgroundToward static test flakiness prediction: a feasibility study
Pontillo, V., Palomba, F., & Ferrucci, F. (2021) · doi:10.1145/3472674.3473981 · Cited by 21
FlakeRepro: automated and efficient reproduction of concurrency-related flaky tests
Leesatapornwongsa, T., Ren, X., & Nath, S. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558956 · Cited by 18
“As aforementioned, there are several studies and reports on flaky tests in both open source and industry [3, 9, 11, 12, 20, 34].”
BackgroundTest Flakiness Across Programming Languages
Barbosa, K., Ferreira, R., Pinto, G., d’Amorim, M., & Miranda, B. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3208864 · Cited by 17
“[74] recently conducted a longitudinal study to observe when –during evolution– flakiness could be observed in tests.”
BackgroundStatic test flakiness prediction: How Far Can We Go?
Pontillo, V., Palomba, F., & Ferrucci, F. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10227-1 · Cited by 17
“…to the body of knowledge through empirical investigations aiming at eliciting the causes of flakiness (Eck et al. 2019; Lam et al. 2020; Lam et al. 2020; Luo et al. 2014; Memon and Cohen 2013) as well as with the definition of techniques for detecting and addressing them (Bell et al.…”
BackgroundFlakeFlagger: Predicting Flakiness Without Rerunning Tests
Alshammari, A., Morris, C., Hilton, M. L., & Bell, J. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion52605.2021.00081 · Cited by 17
Self-citationWei, A., Yi, P., Xie, T., Marinov, D., & Lam, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72016-2_15 · Cited by 16
“For the re-mainder of the paper, we use the 121 modules in which our recent work [25] found at least one flaky test (but not necessarily OD test).”
BackgroundMethodResultInfluentialSelf-citationWhat We Talk About When We Talk About Software Test Flakiness
Barboni, M., Bertolino, A., & Angelis, G. D. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-85347-1_3 · Cited by 16
Software test results exploration and visualization with continuous integration and nightly testing
Strandberg, P. E., Afzal, W., & Sundmark, D. (2022) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-022-00647-1 · Cited by 14
“The first anomaly concerns flaky tests—this is a hot research area and three interesting topics involve: root causes for flaky tests [59], when tests become flaky [25], and if we can predict if tests are to be flaky [34].”
BackgroundEvolution-aware detection of order-dependent flaky tests
Li, C., & Shi, A. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534404 · Cited by 13
“We evaluate IncIDFlakies on a dataset of 67 order-dependent flaky tests where prior work has already identified the code version/commit where the test became flaky, termed the flakinessintroducing commit [27].”
MethodInfluentialWhat Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness
Habchi, S., Haben, G., Sohn, J., Franci, A., Papadakis, M., Cordy, M., & Traon, Y. L. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00039 · Cited by 13
Cost of Flaky Tests in Continuous Integration: An Industrial Case Study
Leinen, F., Elsner, D., Pretschner, A., Stahlbauer, A., Sailer, M., & Jürgens, E. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00037 · Cited by 12
“The existing studies either provide a qualitative overview of the life cycle of flaky tests [12, 14, 21, 22, 23, 24, 25, 26, 27, 28] or technical solutions for one phase of the life cycle [15, 16, 17, 18, 29, 30, 31, 32, 33, 34, 35].”
MethodA Case for Feminism in Programming Language Design
Hermans, F., & Schlesinger, A. (2024) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3689492.3689809 · Cited by 17
Zampetti, F., Nardone, V., & Penta, M. D. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3527948 · Cited by 11
“Flakiness is a relevant problem in testing conventional systems [18, 27, 35, 47, 49].”
BackgroundNeurosymbolic Repair of Test Flakiness
Chen, Y., & Jabbarvand, R. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680369 · Cited by 11
“Previous studies [10, 30] Once the LLM responds to the prompt with a patch, Tailor first checks for compilation errors, which are inevitable in the code produced by LLMs.”
FlakeSync: Automatically Repairing Async Flaky Tests
Rahman, S., & Shi, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639115 · Cited by 10
“Separate groups conducted additional studies to better understand flaky tests [13, 18, 22, 24, 25] with similar findings on top causes.”
BackgroundiPFlakies: A Framework for Detecting and Fixing Python Order-Dependent Flaky Tests
Wang, R., Chen, Y., & Lam, W. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion55297.2022.9793801 · Cited by 10
Self-citationTest Code Flakiness in Mobile Apps: The Developer’s Perspective
Pontillo, V., Palomba, F., & Ferrucci, F. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2023.107394 · Cited by 7
A Large-Scale Empirical Investigation Into Cross-Project Flaky Test Prediction
Afeltra, A., Cannavale, A., Pecorelli, F., Pontillo, V., & Palomba, F. (2024) · IEEE Access · doi:10.1109/access.2024.3458184 · Cited by 7
“For these reasons, researchers have expanded knowledge through empirical studies uncovering causes of flaki-ness [13], [29], [30], [32], [34], and by developing automated techniques to detect and mitigate its impact [4], [12], [50], [53].”
BackgroundAn Empirical Study of Web Flaky Tests: Understanding and Unveiling DOM Event Interaction Challenges
Pei, Y., Sohn, J., & Papadakis, M. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989030 · Cited by 7
“Therefore, the authors confirm that the categorizations of flaky tests proposed by previous studies applied to open-source projects also apply to the proprietary Microsoft projects studied [3].”
Saieva, A., & Kaiser, G. E. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111381 · Cited by 7
Do Test and Environmental Complexity Increase Flakiness? An Empirical Study of SAP HANA
Berndt, A., Bach, T., & Baltes, S. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3695407 · Cited by 6
“Fixing flaky tests is often a tedious task for developers, as flaky failures can be difficult to reproduce [24, 25].”
BackgroundAdaptive Test Healing using LLM/GPT and Reinforcement Learning
Mani, N., & Attaranasl, S. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962516 · Cited by 6
“Large Language Models (LLMs) have revolutionized natural language processing (NLP) by demonstrating exceptional capabilities in understanding and generating human-like text [8].”
Transforming Test Suites into Croissants
Chen, Y., Yildiz, A., Marinov, D., & Jabbarvand, R. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598119 · Cited by 5
“Test-flakiness detection techniques [3, 16, 30, 34–38, 41, 45, 52, 53, 57, 62, 63, 66, 67, 70] rely on several executions of the test suites (same order or different orders of the tests) to observe flaky behavior or search for some pre-defined or learned anti-patterns in the test code to identify…”
BackgroundMethodInfluentialSelf-citationNon-Flaky and Nearly Optimal Time-Based Treatment of Asynchronous Wait Web Tests
Pei, Y., Sohn, J., Habchi, S., & Papadakis, M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695989 · Cited by 5
“Flaky tests are quite common in modern software systems [9, 15, 19, 22, 27, 39].”
Ranking Relevant Tests for Order-Dependent Flaky Tests
Rahman, S., Chanumolu, B. N., Rafi, S., Shi, A., & Lam, W. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00178 · Cited by 4
Self-citationReducing Test Runtime by Transforming Test Fixtures
Li, C., Baz, A., & Shi, A. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695541 · Cited by 3
“We find all polluters for each victim (Line 12) by running each test before the victim one-by-one with the victim to see whether victim fails [28].”
Flaky Tests in UI: Understanding Causes and Applying Correction Strategies
Sousa, É. M., Bezerra, C. I. M., & Machado, I. (2023) · Brazilian Symposium on Software Engineering · doi:10.1145/3613372.3613406 · Cited by 3
“However, this debugging process can be time-consuming and demanding, requiring substantial effort from developers and testers alike [12].”
BackgroundSemantic‐aware two‐phase test case prioritization for continuous integration
Li, Y., Wang, Z., Wang, J., Chen, J., Mou, R., & Li, G. (2023) · Software Testing Verification and Reliability · doi:10.1002/stvr.1864 · Cited by 8
“The flaky tests have gained wide attention in the CI field [37, 38].”
BackgroundCan ChatGPT Repair Non-Order-Dependent Flaky Tests?
Chen, Y., & Jabbarvand, R. (2024) · doi:10.1145/3643656.3643900 · Cited by 3
CI/CD Best Practices: Building Reliable Pipelines
Damarapati, A. (2025) · European Journal of Computer Science and Information Technology · doi:10.37745/ejcsit.2013/vol13n10110 · Cited by 2
A Preliminary Study of Fixed Flaky Tests in Rust Projects on GitHub
Schroeder, T., Phan, M., & Chen, Y. (2025) · doi:10.1109/ftw66604.2025.00010 · Cited by 2
Enhancing Adaptive Test Healing with Graph Neural Networks for Dependency-Aware Decision Making
Mani, N., & Attaranasl, S. (2025) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest66680.2025.00023 · Cited by 2
“Industry reports consistently estimate that flaky tests consume between 5% and 10% of developer time , while also contributing significantly to cloud resource waste, delayed releases, and loss of confidence in test results [2], [3], [4].”
HiFlaky: Hierarchy-aware flakiness classification
Li, Z., Wu, Z., Lei, Y., Xie, H., Li, M., & Hu, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112741 · Cited by 2
A survey of Detecting Flakiness in Automated Test Regression Suite
Elgazzar, M. M. A., Hossny, E., & Omara, F. A. (2024) · Learning and Technology Conference (L&T) · doi:10.1109/lt60077.2024.10469624 · Cited by 2
“One reference empirically investigated whether it was preferable to utilize them to obtain a good detection ratio to run time cost given the possibly prohibitive price of continuously applying automatic flaky test detection techniques [24].”
MethodGraphQL-Aware Healing in Service-Oriented Architectures via Multi-Signal Learning
Mani, N., Attaranasl, S., & He, S. (2025) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose67019.2025.00021 · Cited by 1
Addressing Test Flakiness: Practical Approaches in a Database-Reliant Industrial System
Vegelien, G., Brandt, C., Bas Graaf, A., & Arie Van Deursen, A. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786891 · Cited by 1
“Test data pollution denotes any residual state left behind by one test that influences subsequent tests; polluted database records or global configuration flags have been shown to undermine CI reliability in industrial studies by Bell, Marinov and others [11, 15, 21].”
Static Test Flakiness Prediction
Pontillo, V. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion55297.2022.9793797 · Cited by 1
Regression-Test History Data for Flaky-Test Research
Wendler, P., & Winter, S. (2024) · doi:10.1145/3643656.3643901 · Cited by 1
“We construct a dataset of possible flaky regression-test histories based on IDoFT, as it contains information for “flakiness-introducing commits” (FICs) and commits on which tests have originally been identified as flaky (“iDFlakies commit” [4]) for a subset of flaky tests in the dataset.”
BackgroundMethodSelf-citationTackling Flaky Tests: Understanding the Problem and Providing Practical Solutions
Gruber, M. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678701 · Cited by 1
“already flaky in the commit in which the test was introduced [3], [7], [8].”
BackgroundDiscerning Legitimate Failures From False Alerts: A Study of Chromium's Continuous IntegrationPreprint
Haben, G., Habchi, S., Papadakis, M., Cordy, M., & Traon, Y. L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03382 · Cited by 1
“Many other companies reported having trouble dealing with flakiness, including Microsoft [18], Spotify [22] and Firefox [24].”
InfluentialTest Code Flakiness in Mobile Apps: The Developer's PerspectivePreprint
Pontillo, V., Palomba, F., & Ferrucci, F. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4536718 · Cited by 1
Zhu, J., Long, T., Wang, W., & Memon, A. M. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558941
“Because randomness of the individual pass/fail decisions can arise from ML fuzziness [14] and infrastructure flakiness [20, 42], the holistic measure is effectively an observed statistic based on the specific test input set.”
MethodAnalisis Tes Flakiness Untuk Machine Learning Menggunakan Python
FardilaSoliha, S. (2023) · Teknik Jurnal Ilmu Teknik dan Informatika · doi:10.51903/teknik.v1i1.323
Exploring Tools for Flaky Test Detection, Correction, and Mitigation: A Systematic Mapping Study
Martins, P. A. C., Alves, V. A., de Lima, I. K., Bezerra, C. I. M., & Machado, I. (2024) · doi:10.5753/sast.2024.3700
Efficient Detection of Test Interference in C Projects
Eder, F., & Winter, S. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694995
Self-citationMai, P. T., Son, N. T., Giang, V. T., Viet, B. H., Hai, L. N., & Tung, D. T. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3695
Ivanova, S. (2026) · American Impact Review · doi:10.66308/air.e2026059
Oszkó, M., Liu, X., Zaidman, A., & Brandt, C. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805602
“While prior work on test flakiness often recommends 10 repetitions to estimate failure probabilities [11], our focus is on energy consumption trends rather than precise flakiness modeling.”
Profiling-Guided Bayesian Optimization of JVM Configurations
Baz, A., Lam, W., & Shi, A. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832253
“However, regression testing often imposes substantial time costs on development workflows, creating bottlenecks that slow down release cycles and reduce developer productivity [15, 24, 30, 33, 39, 40, 51].”
Self-citation
Metadata
- Authors
- Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., & Bell, J.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Technische Universität Darmstadt (DE)
- Peking University (CN)
- Northeastern University (US)
- Venue
- Proceedings of the ACM on Programming Languages
- Published
- 2020-11-13
- Volume
- 4
- Issue
- OOPSLA
- Pages
- 1-29
- ISSN
- 2475-1421 (ISSN-L)
- OpenAlex
- W3110368543
- Cited by
- 64
- References
- 37
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 15 · 2023: 6 · 2022: 18 · 2021: 11
- FWCI
- 16.08
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Programming Languages (publishedVersion, cc-by)
- Funders
- National Science Foundation (CNS-1564274, CCF-1763788, CCF-1763822, CCF-1816615, CCF-1844880)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- flaky tests, regression testing, Java projects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
