The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests
Abstract
Regression testing is increasingly important with the wide use of continuous integration. A desirable requirement for regression testing is that a test failure reliably indicates a problem in the code under test and not a false alarm from the test code or the testing infrastructure. However, some test failures are unreliable, stemming from flaky tests that can nondeterministically pass or fail for the same code under test. There are many types of flaky tests, with order-dependent tests being a prominent type. To help advance research on flaky tests, we present (1) a framework, iDFlakies, to detect and partially classify flaky tests; (2) a dataset of flaky tests in open-source projects; and (3) a study with our dataset. iDFlakies automates experimentation with our tool for Maven-based Java projects. Using iDFlakies, we build a dataset of 422 flaky tests, with 50.5% order-dependent and 49.5% not. Our study of these flaky tests finds the prevalence of two types of flaky tests, probability of a test-suite run to have at least one failure due to flaky tests, and how different test reorderings affect the number of detected flaky tests. We envision that our work can spur research to alleviate the problem of flaky tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Cited by
A large-scale longitudinal study of flaky tests
Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., & Bell, J. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428270 · Cited by 64
“…and blog posts from Google [AvoidingFlakeyTests 2019; FlakinessDashboardHOWTO 2020; Micco 2020; Ziftci and Reardon 2017] and Microsoft [Harry 2019; Lam et al. 2019a, 2020a] have reported several challenges with flaky tests, even estimating the monetary cost of flaky tests on developer…”
BackgroundMethodResultInfluentialSelf-citationiPFlakies: a framework for detecting and fixing python order-dependent flaky tests
Wang, R., Chen, Y., & Lam, W. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3516846 · Cited by 36
Self-citationFlakiness Repair in the Era of Large Language Models
Chen, Y. (2024) · Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3639478.3641227 · Cited by 4
Metadata
- Authors
- Lam, W., Oei, R., Shi, A., Marinov, D., & Xie, T.
- Venue
- 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)
- Publisher
- IEEE
- Published
- 2019-04
- Pages
- 312-322
- OpenAlex
- W2951170114
- Cited by
- 162
- References
- 1
- FWCI
- 13.81
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests. https://app.cochat.ai/papers/10.1109/icst.2019.00038
- The paper
- Lam, W., Oei, R., Shi, A., Marinov, D., & Xie, T. (2019). iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests. 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). https://doi.org/10.1109/icst.2019.00038
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.