The DOI matches its Crossref record; 193 of the 193 references it cites were found in a registry or an index.
Mutation Testing Advances: An Analysis and Survey
Open accessAbstract
Mutation testing realizes the idea of using artificial defects to support testing activities. Mutation is typically used as a way to evaluate the adequacy of test suites, to guide the generation of test cases, and to support experimentation. Mutation has reached a maturity phase and gradually gains popularity both in academia and in industry. This chapter presents a survey of recent advances, over the past decade, related to the fundamental problems of mutation testing and sets out the challenges and open problems for the future development of the method. It also collects advices on best practices related to the use of mutation in empirical studies of software testing. Thus, giving the reader a “mini-handbook”-style roadmap for the application of mutation testing as experimental methodology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs
Cadar, C., Dunbar, D., & Engler, D. (2008) · doi:10.5555/1855741.1855756 · Cited by 2,620
Hints on Test Data Selection: Help for the Practicing Programmer
DeMillo, R. A., Lipton, R. J., & Sayward, F. G. (1978) · Computer · doi:10.1109/c-m.1978.218136 · Cited by 2,081
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
Testing Object-Oriented Systems: Models, Patterns, and Tools
Binder, R. V. (1999) · Cited by 1,285
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
GenProg: A Generic Method for Automatic Software Repair
Le Goues, C., Nguyen, T., Forrest, S., & Weimer, W. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,085
Introduction to Software Testing
(2021) · Essentials of Software Testing · doi:10.1017/9781108974073.004 · Cited by 1,056
Arcuri, A., & Briand, L. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985795 · Cited by 968
Inferring causal impact using Bayesian structural time-series models
Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015) · The Annals of Applied Statistics · doi:10.1214/14-aoas788 · Cited by 961
Constraint-based automatic test data generation
DeMilli, R. A., & Offutt, A. J. (1991) · IEEE Transactions on Software Engineering · doi:10.1109/32.92910 · Cited by 804
Is mutation an appropriate tool for testing experiments?
Andrews, J. H., Briand, L. C., & Labiche, Y. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062530 · Cited by 777
Automatically finding patches using genetic programming
Weimer, W., Nguyen, T., Le Goues, C., & Forrest, S. (2009) · 2009 IEEE 31st International Conference on Software Engineering · doi:10.1109/icse.2009.5070536 · Cited by 761
An orchestrated survey of methodologies for automated software test case generation
Anand, S., Burke, E. K., Chen, T. Y., Clark, J., Cohen, M. B., Grieskamp, W., Harman, M., Harrold, M. J., McMinn, P., Bertolino, A., et al. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.02.061 · Cited by 668
An experimental determination of sufficient mutant operators
Offutt, A. J., Lee, A., Rothermel, G., Untch, R. H., & Zapf, C. (1996) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/227607.227610 · Cited by 650
MuJava: an automated class mutation system
Ma, Y.-S., Offutt, J., & Kwon, Y. R. (2005) · Software Testing, Verification and Reliability · doi:10.1002/stvr.308 · Cited by 627
Test case prioritization: an empirical study
Rothermel, G., Untch, R. H., Chu, C., & Harrold, M. J. (1999) · Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360) · doi:10.1109/icsm.1999.792604 · Cited by 625
Introduction to Software Testing
Ammann, P., & Offutt, J. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316771273 · Cited by 607
Requirements Coverage as an Adequacy Measure for Conformance Testing
Rajan, A., Whalen, M. W., Staats, M., & Heimdahl, M. P. E. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88194-0_8 · Cited by 605
Are mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Testing Programs with the Aid of a Compiler
Hamlet, R. G. (1977) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1977.231145 · Cited by 552
Toward a theory of test data selection
Goodenough, J. B., & Gerhart, S. L. (1975) · ACM SIGPLAN Notices · doi:10.1145/390016.808473 · Cited by 514
Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria
Andrews, J. H., Briand, L. C., Labiche, Y., & Namin, A. S. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.83 · Cited by 481
Weak Mutation Testing and Completeness of Test Sets
Howden, W. E. (1982) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1982.235571 · Cited by 477
Automatic patch generation learned from human-written patches
Kim, D., Nam, J., Song, J., & Kim, S. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606626 · Cited by 449
Automatic patch generation learned from human-written patches
Kim, D., Nam, J., Song, J., & Kim, S. (2013) · Cited by 438
Coverage is not strongly correlated with test suite effectiveness
Inozemtseva, L., & Holmes, R. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568271 · Cited by 407
A fortran language system for mutation‐based software testing
King, K. N., & Offutt, A. J. (1991) · Software: Practice and Experience · doi:10.1002/spe.4380210704 · Cited by 379
Mutation 2000: Uniting the Orthogonal
Offutt, A. J., & Untch, R. H. (2001) · Springer eBooks · doi:10.1007/978-1-4757-5939-6_7 · Cited by 374
Metallaxis‐FL: mutation‐based fault localization
Papadakis, M., & Le Traon, Y. (2015) · Software Testing, Verification and Reliability · doi:10.1002/stvr.1509 · Cited by 371
Investigations of the software testing coupling effect
Offutt, A. J. (1992) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/125489.125473 · Cited by 371
Guided, stochastic model-based GUI testing of Android apps
Su, T., Meng, G., Chen, Y., Wu, K., Yang, W., Yao, Y., Pu, G., Liu, Y., & Su, Z. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106298 · Cited by 371
Is mutation an appropriate tool for testing experiments?
Andrews, J. H., Brand, L. C., & Labiche, Y. (2005) · Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. · doi:10.1109/icse.2005.1553583 · Cited by 368
Staged program repair with condition synthesis
Long, F., & Rinard, M. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786811 · Cited by 348
A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each
Le Goues, C., Dewey-Vogt, M., Forrest, S., & Weimer, W. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227211 · Cited by 326
Toward a theory of test data selection
Goodenough, J. B., & Gerhart, S. L. (1975) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1975.6312836 · Cited by 320
An experimental comparison of the effectiveness of branch testing and data flow testing
Frankl, P. G., & Weiss, S. N. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.238581 · Cited by 295
Ask the Mutants: Mutating Faulty Programs for Fault Localization
Moon, S., Kim, Y., Kim, M., & Yoo, S. (2014) · 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2014.28 · Cited by 291
Jia, Y., & Harman, M. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.04.016 · Cited by 287
On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques
Do, H., & Rothermel, G. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.92 · Cited by 287
A theory of fault-based testing
Morell, L. J. (1990) · IEEE Transactions on Software Engineering · doi:10.1109/32.57623 · Cited by 285
Mutation-Driven Generation of Unit Tests and Oracles
Fraser, G., & Zeller, A. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.93 · Cited by 283
Software fault injection: inoculating programs against errors
Voas, J. M., & McGraw, G. E. (1997) · John Wiley & Sons, Inc. eBooks · Cited by 275
PIT: a practical mutation testing tool for Java (demo)
Coles, H., Laurent, T., Henard, C., Papadakis, M., & Ventresque, A. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2948707 · Cited by 270
Leveraging program equivalence for adaptive program repair: Models and first results
Weimer, W., Fry, Z. P., & Forrest, S. (2013) · 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693094 · Cited by 266
Search Based Software Engineering: Techniques, Taxonomy, Tutorial
Harman, M., McMinn, P., de Souza, J. T., & Yoo, S. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25231-0_1 · Cited by 245
Interface Mutation: an approach for integration testing
Delamaro, M. E., Maidonado, J. C., & Mathur, A. P. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.910859 · Cited by 243
All-uses vs mutation testing: An experimental comparison of effectiveness
Frankl, P. G., Weiss, S. N., & Hu, C. (1997) · Journal of Systems and Software · doi:10.1016/s0164-1212(96)00154-9 · Cited by 233
JPF–SE: A Symbolic Execution Extension to Java PathFinder
Anand, S., Păsăreanu, C. S., & Visser, W. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-71209-1_12 · Cited by 232
Genetic Improvement of Software: A Comprehensive Survey
Petke, J., Haraldsson, S. O., Harman, M., Langdon, W. B., White, D. R., & Woodward, J. R. (2018) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2017.2693219 · Cited by 228
Using Mutation to Automatically Suggest Fixes for Faulty Programs
Debroy, V., & Wong, W. E. (2010) · 2010 Third International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2010.66 · Cited by 226
The major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
MuJava: a mutation system for java
Ma, Y.-S., Offutt, J., & Kwon, Y.-R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134425 · Cited by 204
Papadakis, M., Jia, Y., Harman, M., & Traon, Y. L. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.103 · Cited by 202
Constructing Subtle Faults Using Higher Order Mutation Testing
Jia, Y., & Harman, M. (2008) · 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation · doi:10.1109/scam.2008.36 · Cited by 195
A Mutation/Injection-Based Automatic Framework for Evaluating Code Clone Detection Tools
Roy, C. K., & Cordy, J. R. (2009) · 2009 International Conference on Software Testing, Verification, and Validation Workshops · doi:10.1109/icstw.2009.18 · Cited by 193
An empirical evaluation of weak mutation
Offutt, A. J., & Lee, S. D. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.286422 · Cited by 192
Mutation analysis using mutant schemata
Untch, R. H., Offutt, A. J., & Harrold, M. J. (1993) · Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis · doi:10.1145/154183.154265 · Cited by 187
Sufficient mutation operators for measuring test effectiveness
Siami Namin, A., Andrews, J. H., & Murdoch, D. J. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368136 · Cited by 178
Madeyski, L., Orzeszyna, W., Torkar, R., & Jozala, M. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.44 · Cited by 176
An Experimental Evaluation of Data Flow and Mutation Testing
Offutt, A. J., Pan, J., Tewary, K., & Zhang, T. (1996) · Software Practice and Experience · Cited by 175
Code coverage for suite evaluation by developers
Gopinath, R., Jensen, C., & Groce, A. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568278 · Cited by 171
Efficient mutation testing by checking invariant violations
Schüler, D., Dallmeier, V., & Zeller, A. (2009) · doi:10.1145/1572272.1572282 · Cited by 162
Mutation-driven generation of unit tests and oracles
Fraser, G., & Zeller, A. (2010) · International Symposium on Software Testing and Analysis · doi:10.1145/1831708.1831728 · Cited by 159
A study of equivalent and stubborn mutation operators using human analysis of equivalence
Yao, X., Harman, M., & Jia, Y. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568265 · Cited by 159
Chekam, T. T., Papadakis, M., Traon, Y. L., & Harman, M. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.61 · Cited by 154
Mutation analysis testing for finite state machines
Fabbri, S., Delamaro, M. E., Maldonado, J. C., & Masiero, P. C. (2002) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.1994.341378 · Cited by 147
Is operator-based mutant selection superior to random mutant selection?
Zhang, L., Hou, S., Hu, J.-J., Xie, T., & Mei, H. (2010) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1806799.1806863 · Cited by 145
Strong higher order mutation-based test data generation
Harman, M., Jia, Y., & Langdon, W. B. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025144 · Cited by 142
The influence of size and coverage on test suite effectiveness
Namin, A. S., & Andrews, J. H. (2009) · doi:10.1145/1572272.1572280 · Cited by 134
Efficient multi-objective higher order mutation testing with genetic programming
Langdon, W. B., Harman, M., & Jia, Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.07.027 · Cited by 133
Chandra, S., Torlak, E., Barman, S., & Bodík, R. (2011) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/1985793.1985811 · Cited by 132
Arcuri, A., & Briand, L. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001452 · Cited by 130
Are mutation scores correlated with real fault detection?
Papadakis, M., Shin, D., Yoo, S., & Bae, D. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180183 · Cited by 130
Achieving scalable mutation-based generation of whole test suites
Fraser, G., & Arcuri, A. (2014) · Empirical Software Engineering · doi:10.1007/s10664-013-9299-z · Cited by 128
Automatic Mutation Test Case Generation via Dynamic Symbolic Execution
Papadakis, M., & Malevris, N. (2010) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2010.38 · Cited by 127
Li, N., Praphamontripong, U., & Offutt, J. (2009) · International Conference on Software Testing, Verification and Validation Workshops · doi:10.1109/icstw.2009.30 · Cited by 126
Tuya, J., Suárez‐Cabal, M. J., & de la Riva, C. (2006) · Information and Software Technology · doi:10.1016/j.infsof.2006.06.009 · Cited by 124
Threats to the validity of mutation-based test assessment
Papadakis, M., Henard, C., Harman, M., et al. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931040 · Cited by 118
Test generation via Dynamic Symbolic Execution for mutation testing
Zhang, L., Xie, T., Zhang, L., Tillmann, N., de Halleux, J., & Mei, H. (2010) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2010.5609672 · Cited by 117
Using Mutants to Locate "Unknown" Faults
Papadakis, M., & Traon, Y. L. (2012) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2012.159 · Cited by 110
An experimental mutation system for Java
Offutt, J., Ma, Y., & Kwon, Y. R. (2004) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1022494.1022537 · Cited by 108
Detecting Trivial Mutant Equivalences via Compiler Optimisations
Kintis, M., Papadakis, M., Jia, Y., Malevris, N., Traon, Y. L., & Harman, M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2684805 · Cited by 104
Theory of fault-based predicate testing for computer programs
Tai, K. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.536956 · Cited by 101
Faster mutation testing inspired by test prioritization and reduction
Zhang, L., Marinov, D., & Sarfraz Khurshid, D. (2013) · International Symposium on Software Testing and Analysis · doi:10.1145/2483760.2483782 · Cited by 101
Evaluating Mutation Testing Alternatives: A Collateral Experiment
Kintis, M., Papadakis, M., & Malevris, N. (2010) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2010.42 · Cited by 101
Operator-based and random mutant selection: Better together
Zhang, L., Gligoric, M., Marinov, D., & Sarfraz Khurshid, D. (2013) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693070 · Cited by 99
Efficient mutation analysis by propagating and partitioning infected execution states
Just, R., Ernst, M. D., & Fraser, G. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610388 · Cited by 97
Mutation Testing for Aspect-Oriented Programs
Ferrari, F. C., Maldonado, J., & Rashid, A. (2008) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2008.37 · Cited by 96
Mutation Operators for Concurrent Java (J2SE 5.0)
Bradbury, J. S., Cordy, J. R., & Juergen Dingel, A. (2006) · doi:10.1109/mutation.2006.10 · Cited by 96
General test result checking with log file analysis
Andrews, J. H., & Zhang, Y. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1214327 · Cited by 92
A systematic review on search based mutation testing
Silva, R. A., Souza, S. R. S., & de Souza, P. S. L. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.01.017 · Cited by 90
Mutation operators for testing Android apps
Deng, L., Offutt, J., Ammann, P., & Mirzaei, N. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.012 · Cited by 86
Analyzing the validity of selective mutation with dominator mutants
Kurtz, B., Ammann, P., Offutt, J., Delamaro, M. E., Kurtz, M., & Gökçe, N. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2950322 · Cited by 86
Enabling mutation testing for Android apps
Linares‐Vásquez, M., Bavota, G., Tufano, M., Moran, K., Penta, M. D., Vendome, C., Bernal-Cárdenas, C., & Poshyvanyk, D. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106275 · Cited by 84
Automated test suite generation for time-continuous simulink models
Matinnejad, R., Nejati, S., Briand, L., & Bruckmann, T. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884797 · Cited by 82
A mutation carol: Past, present and future
Offutt, J. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.03.007 · Cited by 81
Proteum/IM 2.0: An Integrated Mutation Testing Environment
Delamaro, M. E., Maldonado, J. C., & Vincenzi, A. M. R. (2001) · Springer eBooks · doi:10.1007/978-1-4757-5939-6_17 · Cited by 80
Zhang, J. M., Wang, Z., Zhang, L., Hao, D., Zang, L., Cheng, S., & Zhang, L. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931038 · Cited by 79
A Manifesto for Higher Order Mutation Testing
Harman, M., Jia, Y., & Langdon, W. B. (2010) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2010.13 · Cited by 77
Mutation-Based Fault Localization for Real-World Multilingual Programs (T)
Hong, S., Lee, B., Kwak, T., Jeon, Y., Ko, B., Kim, Y., & Kim, M. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.14 · Cited by 77
Mutation-based test-case prioritization in software evolution
Lou, Y., Hao, D., & Zhang, L. (2015) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2015.7381798 · Cited by 76
Injecting mechanical faults to localize developer faults for evolving software
Zhang, L., Zhang, L., & Sarfraz Khurshid, D. (2013) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2509136.2509551 · Cited by 75
Henard, C., Papadakis, M., Perrouin, G., Klein, J., & Traon, Y. L. (2013) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2013.30 · Cited by 74
Zhang, L., Marinov, D., Zhang, L., & Sarfraz Khurshid, D. (2012) · International Symposium on Software Testing and Analysis · doi:10.1145/2338965.2336793 · Cited by 74
Kintis, M., Papadakis, M., Papadopoulos, A., Valvis, E., Malevris, N., & Traon, Y. L. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9582-5 · Cited by 73
Time for Mutants — Model-Based Mutation Testing with Timed Automata
Aichernig, B. K., Lorber, F., & Ničković, D. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38916-0_2 · Cited by 72
Kaminski, G., Ammann, P., & Offutt, J. (2012) · Journal of Systems and Software · doi:10.1016/j.jss.2012.08.024 · Cited by 72
Papadakis, M., & Malevris, N. (2011) · Software Quality Journal · doi:10.1007/s11219-011-9142-y · Cited by 70
Generating parameterized unit tests
Fraser, G., & Zeller, A. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001464 · Cited by 70
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.31 · Cited by 69
relifix: Automated Repair of Software Regressions
Tan, S. H., & Roychoudhury, A. (2015) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2015.65 · Cited by 69
Mutation based test case generation via a path selection strategy
Papadakis, M., & Malevris, N. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.02.004 · Cited by 68
Mutation Operators for Spreadsheets
Abraham, R., & Erwig, M. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.73 · Cited by 68
Test oracle assessment and improvement
Jahangirova, G., Clark, D., Harman, M., & Tonella, P. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931062 · Cited by 68
Staats, M., Gay, G., & Heimdahl, M. P. E. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227132 · Cited by 67
Guided Mutation Testing for JavaScript Web Applications
Mirshokraie, S., Mesbah, A., & Pattabiraman, K. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2371458 · Cited by 66
Estimation and enhancement of real-time software reliability through mutation analysis
Geist, R., Offutt, A. J., & Harris, F. C. (1992) · IEEE Transactions on Computers · doi:10.1109/12.142681 · Cited by 66
The use of mutation in testing experiments and its sensitivity to external threats
Namin, A. S., & Kakarla, S. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001461 · Cited by 65
Code Defenders: A Mutation Testing Game
Rojas, J. M., & Fraser, G. (2016) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2016.43 · Cited by 65
Guidelines for Coverage-Based Comparisons of Non-Adequate Test Suites
Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, M. A., & Marinov, D. (2015) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2660767 · Cited by 64
Designing Deletion Mutation Operators
Delamaro, M. E., Offutt, J., & Ammann, P. (2014) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2014.12 · Cited by 64
Model-based mutation testing—Approach and case studies
Belli, F., Budnik, C. J., Hollmann, A., et al. (2016) · Science of Computer Programming · doi:10.1016/j.scico.2016.01.003 · Cited by 64
Mutation-aware fault prediction
Bowes, D., Hall, T., Harman, M., Jia, Y., Sarro, F., & Wu, F. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931039 · Cited by 64
Effective fault localization via mutation analysis
Papadakis, M., & Traon, Y. L. (2014) · ACM Symposium on Applied Computing · doi:10.1145/2554850.2554978 · Cited by 62
µDroid: an energy-aware mutation testing framework for Android
Jabbarvand, R., & Malek, S. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106244 · Cited by 62
Inferring mutant utility from program context
Just, R., Kurtz, B., & Ammann, P. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092732 · Cited by 60
Static analysis of mutant subsumption
Kurtz, B., Ammann, P., & Offutt, J. (2015) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2015.7107454 · Cited by 58
Code Defenders: Crowdsourcing Effective Tests and Subtle Mutants with a Mutation Testing Game
Rojas, J. M., White, T. D., Clegg, B., & Fraser, G. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.68 · Cited by 58
Combining mutation and fault localization for automated program debugging
Debroy, V., & Wong, W. E. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.10.042 · Cited by 57
MoMut::UML Model-Based Mutation Testing for UML
Krenn, W., Schlick, R., Tiran, S., Aichernig, B. K., Jöbstl, E., & Brandl, H. (2015) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2015.7102627 · Cited by 55
Finding HTML presentation failures using image comparison techniques
Mahajan, S., & Halfond, W. G. J. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642966 · Cited by 53
Definition of software fault emulation operators: a field data study
Duraes, J. A., & Madeira, H. (2004) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2003.1209922 · Cited by 52
Applying Mutation Testing to Web Applications
Praphamontripong, U., & Offutt, J. (2010) · International Conference on Software Testing, Verification and Validation Workshops · doi:10.1109/icstw.2010.38 · Cited by 52
The care and feeding of wild-caught mutants
Brown, D. B., Vaughn, M. L., Liblit, B., & Reps, T. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106280 · Cited by 49
Sound and Quasi-Complete Detection of Infeasible Test Requirements
Bardin, S., Delahaye, M., David, R., Kosmatov, N., Papadakis, M., Traon, Y. L., & Marion, J.-Y. (2015) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2015.7102607 · Cited by 48
On the number and nature of faults found by random testing
Ciupa, I., Pretschner, A., Oriol, M., Leitner, A., & Meyer, B. (2009) · Software Testing Verification and Reliability · doi:10.1002/stvr.415 · Cited by 48
Experimental Evaluation of SDL and One-Op Mutation for C
Delamaro, M. E., Deng, L., Durelli, V. H. S., Li, N., & Offutt, J. (2014) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2014.33 · Cited by 48
JSEFT: Automated Javascript Unit Test Generation
Mirshokraie, S., Mesbah, A., & Pattabiraman, K. (2015) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2015.7102595 · Cited by 47
Towards automating the generation of mutation tests
Papadakis, M., Malevris, N., & Kallia, M. (2010) · doi:10.1145/1808266.1808283 · Cited by 46
Classifying and Qualifying GUI Defects
Lelli, V., Blouin, A., & Baudry, B. (2015) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2015.7102582 · Cited by 45
Towards mutation analysis of Android apps
Deng, L., Mirzaei, N., Ammann, P., & Offutt, J. (2015) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2015.7107450 · Cited by 44
Time to clean your test objectives
Marcozzi, M., Bardin, S., Kosmatov, N., Papadakis, M., Prévosto, V., & Correnson, L. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180191 · Cited by 41
Automated Generation of Pointcut Mutants for Testing Pointcuts in AspectJ Programs
Anbalagan, P., & Xie, T. (2008) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2008.58 · Cited by 40
Is Branch Coverage a Good Measure of Testing Effectiveness?
Wei, Y., Meyer, B., & Manuel Oriol, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-25231-0_5 · Cited by 39
An Empirical Study on the Scalability of Selective Mutation Testing
Zhang, J. M., Zhu, M., Hao, D., & Zhang, L. (2014) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2014.27 · Cited by 39
Strong mutation-based test data generation using hill climbing
Souza, F. C. M., Papadakis, M., Traon, Y. L., & Delamaro, M. E. (2016) · IEEE/ACM International Workshop on Search-Based Software Testing (SBST) · doi:10.1145/2897010.2897012 · Cited by 39
The Csaw C Mutation Tool: Initial Results
Ellims, M., Ince, D. C., & Petre, M. (2007) · doi:10.1109/taic.part.2007.28 · Cited by 38
Can testedness be effectively measured?
Ahmed, I., Gopinath, R., Brindescu, C., Groce, A., & Jensen, C. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2950324 · Cited by 38
Mutant reduction based on dominance relation for weak mutation testing
Gong, D., Zhang, G., Yao, X., & Meng, F. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.05.001 · Cited by 37
MUSEUM: Debugging real-world multilingual programs using mutation analysis
Hong, S., Kwak, T., Lee, B., Jeon, Y., Ko, B., Kim, Y., & Kim, M. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.10.002 · Cited by 37
A Theoretical and Empirical Study of Diversity-Aware Mutation Adequacy Criterion
Shin, D., Yoo, S., & Bae, D. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2732347 · Cited by 37
Predicate-based test generation for computer programs
Tai, K. (1993) · doi:10.5555/257572.257631 · Cited by 36
Kaminski, G., Ammann, P., & Offutt, J. (2011) · International Workshop on Automation of Software Test (AST) · doi:10.1145/1982595.1982608 · Cited by 34
Using conditional mutation to increase the efficiency of mutation analysis
Just, R., Kapfhammer, G. M., & Schweiggert, F. (2011) · International Workshop on Automation of Software Test (AST) · doi:10.1145/1982595.1982606 · Cited by 34
Using constraints to diagnose faulty spreadsheets
Abreu, R., Hofer, B., Perez, A., & Wotawa, F. (2014) · Software Quality Journal · doi:10.1007/s11219-014-9236-4 · Cited by 34
Mutation at System and Functional Levels
Mateo, P. R., Polo, M., & Offutt, J. (2010) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2010.18 · Cited by 33
Faster mutation-based fault localization with a novel mutation execution strategy
Gong, P., Zhao, R., & Li, Z. (2015) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2015.7107448 · Cited by 32
Efficient Leveraging of Symbolic Execution to Advanced Coverage Criteria
Bardin, S., Kosmatov, N., & Cheynier, F. (2014) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2014.30 · Cited by 32
Whitening SOA Testing via Event Exposure
Ye, C., & Jacobsen, H. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.20 · Cited by 32
MEDIC: A static analysis framework for equivalent mutant identification
Kintis, M., & Malevris, N. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.07.009 · Cited by 31
Inferring Loop Invariants by Mutation, Dynamic Analysis, and Static Checking
Galeotti, J. P., Furia, C. A., May, E., et al. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2431688 · Cited by 31
An Analysis of OO Mutation Operators
Hu, J., Li, N., & Offutt, J. (2011) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2011.47 · Cited by 29
Ramler, R., Wetzlmaier, T., & Klammer, C. (2017) · ACM Symposium on Applied Computing · doi:10.1145/3019612.3019830 · Cited by 29
Searching and generating test inputs for mutation testing
Papadakis, M., & Malevris, N. (2013) · SpringerPlus · doi:10.1186/2193-1801-2-121 · Cited by 28
Applying Mutation Analysis on Kernel Test Suites: An Experience Report
Ahmed, I., Jensen, C., Groce, A., & McKenney, P. E. (2017) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2017.26 · Cited by 26
PYTHIA: Generating test cases with oracles for JavaScript applications
Mirshokraie, S., Mesbah, A., & Pattabiraman, K. (2013) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693121 · Cited by 22
Identifying More Equivalent Mutants via Code Similarity
Kintis, M., & Malevris, N. (2013) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2013.34 · Cited by 22
Papadakis, M., Shin, D., Yoo, S., & Bae, D.-H. (2018) · Open Repository and Bibliography (University of Luxembourg) · Cited by 22
Test Data Generation Techniques for Mutation Testing: A Systematic Mapping
Carlos, F., Souza, M., Papadakis, M., Durelli, V. H. S., & Delamaro, M. E. (2014) · Open Repository and Bibliography (University of Luxembourg) · doi:10.13140/rg.2.1.3699.9209 · Cited by 21
Generating Distinguishing Tests Using the Minion Constraint Solver
Wotawa, F., Nica, M., & Aichernig, B. K. (2010) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2010.11 · Cited by 21
An Effective Path Selection Strategy for Mutation Testing
Papadakis, M., & Malevris, N. (2009) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2009.68 · Cited by 20
Using Data Flow Patterns for Equivalent Mutant Detection
Kintis, M., & Malevris, N. (2014) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2014.21 · Cited by 19
A variability perspective of mutation analysis
Devroey, X., Perrouin, G., Cordy, M., Papadakis, M., Legay, A., & Schobbens, P. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2666610 · Cited by 19
FSX: fine-grained incremental unit test generation for C/C++ programs
Yoshida, H., Tokumoto, S., Prasad, M. R., Ghosh, I., & Uehara, T. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931055 · Cited by 18
Isomorphic regression testing: executing uncovered branches without test augmentation
Zhang, J. M., Lou, Y., Zhang, L., Hao, D., Zhang, L., & Mei, H. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2950313 · Cited by 18
How Verified is My Code? Falsification-Driven Verification (T)
Groce, A., Ahmed, I., Jensen, C., & McKenney, P. E. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.40 · Cited by 17
Nequivack: Assessing Mutation Score Confidence
Holling, D., Bănescu, S., Probst, M., Petrovska, A., & Pretschner, A. (2016) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2016.29 · Cited by 17
Efficient mutation testing of multithreaded code
Gligoric, M., Jagannath, V., Luo, Q., & Marinov, D. (2012) · Software Testing Verification and Reliability · doi:10.1002/stvr.1469 · Cited by 16
Assessing and Comparing Mutation-based Fault Localization TechniquesPreprint
Chekam, T. T., Papadakis, M., & Traon, Y. L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.05512 · Cited by 15
Proteum/FL: A tool for localizing faults using mutation analysis
Papadakis, M., Delamaro, M. E., & Traon, Y. L. (2013) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2013.6648189 · Cited by 14
Reducing Mutants with Mutant Killable Precondition
Iida, C., & Takada, S. (2017) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2017.29 · Cited by 14
Fault domain-based testing in imperfect situations: a heuristic approach and case studies
Belli, F., Beyazıt, M., Endo, A. T., et al. (2014) · Software Quality Journal · doi:10.1007/s11219-014-9242-6 · Cited by 13
An Evaluation of Mutation and Data-Flow Testing: A Meta-analysis
Kakarla, S., Momotaz, S., & Namin, A. S. (2011) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2011.51 · Cited by 12
Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Capability
Shin, D., Yoo, S., & Bae, D. (2016) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2016.37 · Cited by 12
Combining weak and strong mutation for a noninterpretive Java mutation system
Kim, S., Ma, Y., & Kwon, Y. R. (2012) · Software Testing Verification and Reliability · doi:10.1002/stvr.1480 · Cited by 11
Test Case Generation from Mutants Using Model Checking Techniques
Riener, H., Bloem, R., & Fey, G. (2011) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2011.55 · Cited by 11
KD-ART: Should we intensify or diversify tests to kill mutants?
Patrick, M., & Jia, Y. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.009 · Cited by 11
ForkSim: Generating software forks for evaluating cross-project similarity analysis tools
Svajlenko, J., Roy, C. K., & Duszynski, S. (2013) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2013.6648182 · Cited by 10
Automatic mutation based test data generation
Papadakis, M., & Malevris, N. (2011) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/2001858.2001997 · Cited by 10
Kernel Density Adaptive Random Testing
Patrick, M., & Jia, Y. (2015) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2015.7107451 · Cited by 6
Evaluating Automated Unit Testing in Sulu
Tan, R. P., & Edwards, S. H. (2008) · International Conference on Software Testing, Verification, and Validation · doi:10.1109/icst.2008.59 · Cited by 6
Testing stochastic software using pseudo-oracles
Patrick, M. T., Craig, A. P., Cunniffe, N. J., Parry, M. F., & Gilligan, C. A. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931063 · Cited by 6
Cited by
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
Self-citationLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Self-citationDeep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167
Effective test generation using pre-trained Large Language Models and mutation testing
Dakhel, A. M., Nikanjam, A., Majdinasab, V., Khomh, F., & Desmarais, M. C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107468 · Cited by 157
An Autocorrelation-based LSTM-Autoencoder for Anomaly Detection on Time-Series Data
Homayouni, H., Ghosh, S., Ray, I., Gondalia, S., Duggan, J., & Kahn, M. G. (2020) · 2020 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata50022.2020.9378192 · Cited by 66
Metadata
- Venue
- Advances in Computers
- Publisher
- Elsevier
- Published
- 2019
- Pages
- 275-378
- ISSN
- 0065-2458 (ISSN-L)
- OpenAlex
- W2725449579
- Cited by
- 425
- References
- 193
- FWCI
- 230.77
- Type
- book chapter
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.