The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
iPFlakies: a framework for detecting and fixing python order-dependent flaky tests
Abstract
Developers typically run tests after code changes. Flaky tests, which are tests that can nondeterministically pass and fail when run on the same version of code, can mislead developers about their recent changes. Much of the prior work on flaky tests is focused on Java projects. One prominent category of flaky tests that the prior work focused on is order-dependent (OD) tests, which are tests that pass or fail depending on the order in which tests are run. For example, our prior work proposed using other tests in the test suite to fix (or correctly set up) the state needed by Java OD tests to pass.Unlike Java flaky tests, flaky tests in other programming languages have received less attention. To help with this problem, another piece of prior work recently studied flaky tests in Python projects and detected many OD tests. Unfortunately, the work did not identify the other tests in the test suites that can be used to fix the OD tests. To fill this gap, we propose iPFlakies, a framework for automatically detecting and fixing Python OD tests. Using iPFlakies, we extend the prior work’s dataset to include (1) tests that can be used to reproduce and fix the OD tests and (2) patches for the OD tests. Our work to extend the dataset finds that reproducing passing and failing test results of flaky tests can be difficult and that iPFlakies is effective at detecting and fixing Python OD tests. To aid future research, we make our iPFlakies framework, dataset improvements, and experimental infrastructure publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A large-scale longitudinal study of flaky tests
Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., & Bell, J. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428270 · Cited by 64 · cited 2×
“Since the taxonomy, much of the flaky-test work has focused on OD tests [1, 2, 7–9, 12, 15, 19].”
BackgroundSelf-citation
Flagged first, then by citations
Research works printed but not linked: 13
- Martin Gruber Stephan Lukasczyk Florian Kroiß and Gordon Fraser. 2021. An empirical study of flaky tests in Python. In ICST. Martin Gruber Stephan Lukasczyk Florian Kroiß and Gordon Fraser. 2021. An empirical study of flaky tests in Python. In ICST. · DOI
- Alex Gyori August Shi Farah Hariri and Darko Marinov. 2015. Reliable testing: Detecting state-polluting tests to prevent test dependency. In ISSTA. Alex Gyori August Shi Farah Hariri and Darko Marinov. 2015. Reliable testing: Detecting state-polluting tests to prevent test dependency. In ISSTA. · DOI
- Mark Harman and Peter O'Hearn. 2018. From start-ups to scale-ups: Opportunities and open problems for static and dynamic program analysis. In SCAM. Mark Harman and Peter O'Hearn. 2018. From start-ups to scale-ups: Opportunities and open problems for static and dynamic program analysis. In SCAM. · DOI
- Emily Kowalczyk Karan Nair Zebao Gao Leo Silberstein Teng Long and Atif Memon. 2020. Modeling and ranking flaky tests at Apple. In ICSE SEIP. Emily Kowalczyk Karan Nair Zebao Gao Leo Silberstein Teng Long and Atif Memon. 2020. Modeling and ranking flaky tests at Apple. In ICSE SEIP. · DOI
- Wing Lam Kivanç Muşlu Hitesh Sajnani and Suresh Thummalapenta. 2020. A study on the lifecycle of flaky tests. In ICSE. Wing Lam Kivanç Muşlu Hitesh Sajnani and Suresh Thummalapenta. 2020. A study on the lifecycle of flaky tests. In ICSE. · DOI
- Wing Lam Reed Oei August Shi Darko Marinov and Tao Xie. 2019. iDFlakies: A framework for detecting and partially classifying flaky tests. In ICST. Wing Lam Reed Oei August Shi Darko Marinov and Tao Xie. 2019. iDFlakies: A framework for detecting and partially classifying flaky tests. In ICST. · DOI
- Wing Lam Stefan Winter Angello Astorga Victoria Stodden and Darko Marinov. 2020. Understanding reproducibility and characteristics of flaky tests through test reruns in Java projects. In ISSRE. Wing Lam Stefan Winter Angello Astorga Victoria Stodden and Darko Marinov. 2020. Understanding reproducibility and characteristics of flaky tests through test reruns in Java projects. In ISSRE. · DOI
- Chengpeng Li Chenguang Zhu Wenxi Wang and August Shi. 2022. Repairing order-dependent flaky tests via test generation. In ICSE. Chengpeng Li Chenguang Zhu Wenxi Wang and August Shi. 2022. Repairing order-dependent flaky tests via test generation. In ICSE.
- Atif Memon Zebao Gao Bao Nguyen Sanjeev Dhanda Eric Nickell Rob Siemborski and John Micco. 2017. Taming Google-scale continuous testing. In ICSE SEIP. Atif Memon Zebao Gao Bao Nguyen Sanjeev Dhanda Eric Nickell Rob Siemborski and John Micco. 2017. Taming Google-scale continuous testing. In ICSE SEIP. · DOI
- Suchita Mukherjee Abigail Almanza and Cindy Rubio-González. 2021. Fixing dependency errors for Python build reproducibility. In ISSTA. Suchita Mukherjee Abigail Almanza and Cindy Rubio-González. 2021. Fixing dependency errors for Python build reproducibility. In ISSTA. · DOI
- August Shi Wing Lam Reed Oei Tao Xie and Darko Marinov. 2019. iFixFlakies: A framework for automatically fixing order-dependent flaky tests. In ESEC/FSE. August Shi Wing Lam Reed Oei Tao Xie and Darko Marinov. 2019. iFixFlakies: A framework for automatically fixing order-dependent flaky tests. In ESEC/FSE.
- Andreas Zeller and Ralf Hildebrandt . 2002. Simplifying and isolating failure-inducing input. TSE ( 2002 ). Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failure-inducing input. TSE (2002).
- Sai Zhang Darioush Jalali Jochen Wuttke Kıvanç Muşlu Wing Lam Michael D. Ernst and David Notkin. 2014. Empirically revisiting the test independence assumption. In ISSTA. Sai Zhang Darioush Jalali Jochen Wuttke Kıvanç Muşlu Wing Lam Michael D. Ernst and David Notkin. 2014. Empirically revisiting the test independence assumption. In ISSTA. · DOI
Metadata
- Authors
- Wang, R., Chen, Y., & Lam, W.
- Affiliations
- Zhejiang University (CN)
- Huazhong University of Science and Technology (CN)
- George Mason University (US)
- Venue
- Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings
- Publisher
- ACM
- Published
- 2022-05-21
- Pages
- 120-124
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4282842157
- Cited by
- 36
- References
- 1
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 12 · 2023: 8 · 2022: 1
- FWCI
- 5.72
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- flaky tests, Python, test suite ordering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
