The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Coverage is not strongly correlated with test suite effectiveness
Abstract
The coverage of a test suite is often used as a proxy for its ability to detect faults. However, previous studies that investigated the correlation between code coverage and test suite effectiveness have failed to reach a consensus about the nature and strength of the relationship between these test suite characteristics. Moreover, many of the studies were done with small or synthetic programs, making it unclear whether their results generalize to larger programs, and some of the studies did not account for the confounding influence of test suite size. In addition, most of the studies were done with adequate suites, which are are rare in practice, so the results may not generalize to typical test suites.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fundamental Statistics in Psychology and Education.
Smith, B. B., & Guilford, J. P. (1951) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2980908 · Cited by 3,455
Is mutation an appropriate tool for testing experiments?
Andrews, J. H., Briand, L. C., & Labiche, Y. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062530 · Cited by 777
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Quantitative analysis of faults and failures in a complex software system
Fenton, N. E., & Ohlsson, N. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.879815 · Cited by 699
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationUsing Mutation Analysis for Assessing and Comparing Testing Coverage Criteria
Andrews, J. H., Briand, L. C., Labiche, Y., & Namin, A. S. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.83 · Cited by 481
Fundamental statistics in psychology and education
Guilford, J. P., & Fruchter, B. (1973) · Cited by 447
The confounding effect of class size on the validity of object-oriented metrics
El Emam, K., Benlarbi, S., Goel, N., & Rai, S. N. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.935855 · Cited by 422
Investigations of the software testing coupling effect
Offutt, A. J. (1992) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/125489.125473 · Cited by 371
An experimental comparison of the effectiveness of branch testing and data flow testing
Frankl, P. G., & Weiss, S. N. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.238581 · Cited by 295
All-uses vs mutation testing: An experimental comparison of effectiveness
Frankl, P. G., Weiss, S. N., & Hu, C. (1997) · Journal of Systems and Software · doi:10.1016/s0164-1212(96)00154-9 · Cited by 233
Code coverage for suite evaluation by developers
Gopinath, R., Jensen, C., & Groce, A. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568278 · Cited by 171
Nonparametric Measures of Association
Gibbons, J. (1993) · Sage eBooks · doi:10.4135/9781412985291 · Cited by 168
Daran, M., & Thévenod-Fosse, P. (1996) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/226295.226313 · Cited by 154
The influence of size and coverage on test suite effectiveness
Namin, A. S., & Andrews, J. H. (2009) · doi:10.1145/1572272.1572280 · Cited by 134
Detecting equivalent mutants and the feasible path problem
Offutt, A. J., & Pan, J. (2002) · Annual Conference on Computer Assurance, Systems Integrity, Software Safety and Process Security (COMPASS) · doi:10.1109/cmpass.1996.507890 · Cited by 126
Fundamental Statistics in Psychology and Education.
Brumbaugh, M. A., & Guilford, J. P. (1943) · Journal of the American Statistical Association · doi:10.2307/2279562 · Cited by 119
The effect of code coverage on fault detection under different testing profiles
Cai, X., & Lyu, M. R.-T. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083288 · Cited by 80
Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites
Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, M. A., & Marinov, D. (2015) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2660767 · Cited by 64
Marick, B. (1999) · Cited by 53
A Modification of Kendall's Tau for the Case of Arbitrary Ties in Both Rankings
Adler, L. M. (1957) · Journal of the American Statistical Association · doi:10.1080/01621459.1957.10501365 · Cited by 27
Cited by
Klees, G., Ruef, A., Cooper, B., Wei, S., & Hicks, M. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243804 · Cited by 633
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationMutation Testing Advances: An Analysis and Survey
(2019) · Advances in Computers · doi:10.1016/bs.adcom.2018.03.015 · Cited by 425
On the reliability of coverage-based fuzzer benchmarking
Böhme, M., Szekeres, L., & Metzman, J. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510230 · Cited by 104
“The current guideline on sound fuzzer evaluation suggests that coverage-based benchmarking alone may be insufficient (referring to the contentious study [9] by Inozemtseva and Holmes [32] which suggests a weak correlation).”
Background
Metadata
- Authors
- Inozemtseva, L., & Holmes, R.
- Venue
- Proceedings of the 36th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2014-05-31
- Pages
- 435-445
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W2106072155
- Cited by
- 407
- References
- 21
- FWCI
- 28.54
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.