The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Are mutants a valid substitute for real faults in software testing?
Abstract
A good test suite is one that detects real faults. Because the set of faults in a program is usually unknowable, this definition is not useful to practitioners who are creating test suites, nor to researchers who are creating and evaluating tools that generate test suites. In place of real faults, testing research often uses mutants, which are artificial faults -- each one a simple syntactic variation -- that are systematically seeded throughout the program under test. Mutation analysis is appealing because large numbers of mutants can be automatically-generated and used to compensate for low quantities or the absence of known real faults. Unfortunately, there is little experimental evidence to support the use of mutants as a replacement for real faults. This paper investigates whether mutants are indeed a valid substitute for real faults, i.e., whether a test suite’s ability to detect mutants is correlated with its ability to detect real faults that developers have fixed. Unlike prior studies, these investigations also explicitly consider the conflating effects of code coverage on the mutant detection rate. Our experiments used 357 real faults in 5 open-source applications that comprise a total of 321,000 lines of code. Furthermore, our experiments used both developer-written and automatically-generated test suites. The results show a statistically significant correlation between mutant detection and real fault detection, independently of code coverage. The results also give concrete suggestions on how to improve mutation analysis and reveal some inherent limitations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hints on Test Data Selection: Help for the Practicing Programmer
DeMillo, R. A., Lipton, R. J., & Sayward, F. G. (1978) · Computer · doi:10.1109/c-m.1978.218136 · Cited by 2,081
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Self-citationPrioritizing test cases for regression testing
Rothermel, G., Untch, R. H., Chu, C., & Harrold, M. J. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.962562 · Cited by 1,338
Empirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/1101908.1101949 · Cited by 1,238
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
Self-citationTest case prioritization: a family of empirical studies
Elbaum, S., Malishevsky, A. G., & Rothermel, G. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988497 · Cited by 943
Feedback-Directed Random Test Generation
Pacheco, C., Lahiri, S. K., Ernst, M. D., & Ball, T. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.37 · Cited by 863
Self-citationConstraint-based automatic test data generation
DeMilli, R. A., & Offutt, A. J. (1991) · IEEE Transactions on Software Engineering · doi:10.1109/32.92910 · Cited by 804
Is mutation an appropriate tool for testing experiments?
Andrews, J. H., Briand, L. C., & Labiche, Y. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062530 · Cited by 777
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
An experimental determination of sufficient mutant operators
Offutt, A. J., Lee, A., Rothermel, G., Untch, R. H., & Zapf, C. (1996) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/227607.227610 · Cited by 650
JCrasher: an automatic robustness tester for Java
Csallner, C., & Smaragdakis, Y. (2004) · Software: Practice and Experience · doi:10.1002/spe.602 · Cited by 426
Coverage is not strongly correlated with test suite effectiveness
Inozemtseva, L., & Holmes, R. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568271 · Cited by 407
Self-citationInvestigations of the software testing coupling effect
Offutt, A. J. (1992) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/125489.125473 · Cited by 371
An empirical study of the effects of minimization on the fault detection capabilities of test suites
Rothermel, G., Harrold, M. J., Ostrin, J., & Hong, C. (2002) · International Conference on Software Maintenance · doi:10.1109/icsm.1998.738487 · Cited by 303
Jia, Y., & Harman, M. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.04.016 · Cited by 287
Mutation-Driven Generation of Unit Tests and Oracles
Fraser, G., & Zeller, A. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.93 · Cited by 283
Self-citationThe major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationImproving test suites for efficient fault localization
Baudry, B., Fleurey, F., & Le Traon, Y. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134299 · Cited by 194
The missing links: bugs and bug-fix commits
Bachmann, A., Bird, C., Rahman, F., Devanbu, P., & Bernstein, A. (2010) · Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1882291.1882308 · Cited by 190
Sufficient mutation operators for measuring test effectiveness
Siami Namin, A., Andrews, J. H., & Murdoch, D. J. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368136 · Cited by 178
Empirically revisiting the test independence assumption
Zhang, S., Jalali, D., Wuttke, J., Muşlu, K., Lam, W., Ernst, M. D., & Notkin, D. S. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610404 · Cited by 166
Self-citationImproving test suites via operational abstraction
Harder, M. P. F., Mellen, J., & Ernst, M. D. (2003) · doi:10.5555/776816.776824 · Cited by 163
Self-citationA study of equivalent and stubborn mutation operators using human analysis of equivalence
Yao, X., Harman, M., & Jia, Y. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568265 · Cited by 159
Daran, M., & Thévenod-Fosse, P. (1996) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/226295.226313 · Cited by 154
Strong higher order mutation-based test data generation
Harman, M., Jia, Y., & Langdon, W. B. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025144 · Cited by 142
MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler
Just, R., Schweiggert, F., & Kapfhammer, G. M. (2011) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2011.6100138 · Cited by 136
Self-citationAchieving scalable mutation-based generation of whole test suites
Fraser, G., & Arcuri, A. (2014) · Empirical Software Engineering · doi:10.1007/s10664-013-9299-z · Cited by 128
Self-citationAutomatic Mutation Test Case Generation via Dynamic Symbolic Execution
Papadakis, M., & Malevris, N. (2010) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2010.38 · Cited by 127
Test generation via Dynamic Symbolic Execution for mutation testing
Zhang, L., Xie, T., Zhang, L., Tillmann, N., de Halleux, J., & Mei, H. (2010) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2010.5609672 · Cited by 117
Sample size vs. bias in defect prediction
Rahman, F., Posnett, D., Herraiz, I., & Dévanbu, P. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2491418 · Cited by 117
DiffGen: Automated Regression Unit-Test Generation
Taneja, K., & Xie, T. (2008) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2008.60 · Cited by 102
Efficient mutation analysis by propagating and partitioning infected execution states
Just, R., Ernst, M. D., & Fraser, G. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610388 · Cited by 97
Self-citationThe class-level mutants of MuJava
Offutt, J., Ma, Y., & Kwon, Y. R. (2006) · International Workshop on Automation of Software Test (AST) · doi:10.1145/1138929.1138945 · Cited by 92
Automatic recovery from runtime failures
Carzaniga, A., Gorla, A., Mattavelli, A., Perino, N., & Pezzè, M. (2013) · doi:10.5555/2486788.2486891 · Cited by 81
DSD-Crasher: a hybrid analysis tool for bug finding
Csallner, C., & Smaragdakis, Y. (2006) · International Symposium on Software Testing and Analysis · doi:10.1145/1146238.1146267 · Cited by 76
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
Self-citationThe use of mutation in testing experiments and its sensitivity to external threats
Namin, A. S., & Kakarla, S. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001461 · Cited by 65
Cited by
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Self-citationEvaluating and Improving Fault Localization
Pearson, S., Campos, J., Just, R., Fraser, G., Abreu, R., Ernst, M. D., Pang, D., & Keller, B. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.62 · Cited by 426
Self-citationMutation Testing Advances: An Analysis and Survey
(2019) · Advances in Computers · doi:10.1016/bs.adcom.2018.03.015 · Cited by 425
Coverage is not strongly correlated with test suite effectiveness
Inozemtseva, L., & Holmes, R. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568271 · Cited by 407
Self-citationThe ManyBugs and IntroClass benchmarks for automated repair of C programs
Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2454513 · Cited by 326
Metadata
- Authors
- Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G.
- Venue
- Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2014-11-11
- Pages
- 654-665
- OpenAlex
- W2135937266
- Cited by
- 595
- References
- 40
- FWCI
- 34.51
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.