The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
An empirical analysis of flaky tests
Abstract
Regression testing is a crucial part of software development. It checks that software changes do not break existing functionality. An important assumption of regression testing is that test outcomes are deterministic: an unmodified test is expected to either always pass or always fail for the same code under test. Unfortunately, in practice, some tests often called flaky tests—have non-deterministic outcomes. Such tests undermine the regression testing as they make it difficult to rely on test results. We present the first extensive study of flaky tests. We study in detail a total of 201 commits that likely fix flaky tests in 51 open-source projects. We classify the most common root causes of flaky tests, identify approaches that could manifest flaky behavior, and describe common strategies that developers use to fix flaky tests. We believe that our insights and implications can help guide future research on the important topic of (avoiding) flaky tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Learning from mistakes: a comprehensive study on real world concurrency bug characteristics
Lu, S., Park, S., Seo, E., & Zhou, Y. (2008) · Proceedings of the 13th international conference on Architectural support for programming languages and operating systems · doi:10.1145/1346281.1346323 · Cited by 895
Have things changed now?: an empirical study of bug characteristics in modern open source software
Li, Z., Tan, L., Wang, X., Lu, S., Zhou, Y., & Zhai, C. (2006) · Proceedings of the 1st workshop on Architectural and system support for improving software dependability · doi:10.1145/1181309.1181314 · Cited by 291
Characterizing and predicting which bugs get fixed: an empirical study of Microsoft Windows
Guo, P. J., Zimmermann, T., Nagappan, N., & Murphy, B. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806871 · Cited by 265
The missing links: bugs and bug-fix commits
Bachmann, A., Bird, C., Rahman, F., Devanbu, P., & Bernstein, A. (2010) · Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1882291.1882308 · Cited by 190
Empirically revisiting the test independence assumption
Zhang, S., Jalali, D., Wuttke, J., Muşlu, K., Lam, W., Ernst, M. D., & Notkin, D. S. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610404 · Cited by 166
Automated atomicity-violation fixing
Jin, G., Song, L., Zhang, W., Lu, S., & Liblit, B. (2011) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1993498.1993544 · Cited by 151
Automated atomicity-violation fixing
Jin, G., Song, L., Zhang, W., Lu, S., & Liblit, B. (2011) · ACM SIGPLAN Notices · doi:10.1145/1993316.1993544 · Cited by 135
ReAssert: Suggesting Repairs for Broken Unit Tests
Daniel, B., Jagannath, V., Dig, D., & Marinov, D. (2009) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2009.17 · Cited by 110
Self-citationAutomatic detection of floating-point exceptions
Barr, E. T., Vo, T., Le, V., & Su, Z. (2013) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/2429069.2429133 · Cited by 101
Cited by
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336
Shamshiri, S., Just, R., Rojas, J. M., Fraser, G., McMinn, P., & Arcuri, A. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.86 · Cited by 230
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
A large-scale longitudinal study of flaky tests
Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., & Bell, J. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428270 · Cited by 64
“Our literature search reveals only two research papers [Gambi et al. 2018; Luo et al. 2014] that mention checking whether tests are flaky when they are added, but both studies are quite limited to just a handful of flaky tests that were manually studied, and in at least one study [Luo et al. 2014]…”
BackgroundInfluentialSelf-citationiPFlakies: a framework for detecting and fixing python order-dependent flaky tests
Wang, R., Chen, Y., & Lam, W. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3516846 · Cited by 36
Metadata
- Authors
- Luo, Q., Hariri, F., Eloussi, L., & Marinov, D.
- Venue
- Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2014-11-11
- Pages
- 643-653
- OpenAlex
- W2026926213
- Cited by
- 450
- References
- 17
- FWCI
- 8.19
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An empirical analysis of flaky tests. https://app.cochat.ai/papers/10.1145/2635868.2635920
- The paper
- Luo, Q., Hariri, F., Eloussi, L., & Marinov, D. (2014). An empirical analysis of flaky tests. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. https://doi.org/10.1145/2635868.2635920
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
