The DOI matches its Crossref record; 89 of the 93 references it cites were found in a registry or an index.
Machine Learning Applied to Software Testing: A Systematic Mapping Study
Abstract
Software testing involves probing into the behavior of software systems to uncover faults. Most testing activities are complex and costly, so a practical strategy that has been adopted to circumvent these issues is to automate software testing. There has been a growing interest in applying machine learning (ML) to automate various software engineering activities, including testing-related ones. In this paper, we set out to review the state-of-the art of how ML has been explored to automate and streamline software testing and provide an overview of the research at the intersection of these two fields by conducting a systematic mapping study. We selected 48 primary studies. These selected studies were then categorized according to study type, testing activity, and ML algorithm employed to automate the testing activity. The results highlight the most widely used ML algorithms and identify several avenues for future research. We found that ML algorithms have been used mainly for test-case generation, refinement, and evaluation. Also, ML has been used to evaluate test oracle construction and to predict the cost of testing-related activities. The results of this paper outline the ML algorithms that are most commonly used to automate software-testing activities, helping researchers to understand the current state of research concerning ML applied to software testing. We also found that there is a need for better empirical studies examining how ML algorithms have been used to automate software-testing activities.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Test case evaluation is a complex task since it is hard to formalize and measure test cases' characteristics influence quality (Durelli et al., 2019).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Durelli et al., 2019
Citations
Cites
(1963) · Medical Entomology and Zoology · Cited by 15,085
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 2×
“Throughout the backward snowballing process, we followed the guidelines provided by Wohlin [35].”
BackgroundMethodUnderstanding Machine Learning: From Theory to Algorithms
Shalev-Shwartz, S., & Ben-David, S. (2014) · doi:10.1017/cbo9781107298019 · Cited by 3,115
(2015) · Cited by 2,857
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,752
The Goal Question Metric Approach
Basili, V. R., Caldiera, G., & Rombach, H. D. (1994) · Cited by 1,651
Prioritizing test cases for regression testing
Rothermel, G., Untch, R. H., Chu, C., & Harrold, M. J. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.962562 · Cited by 1,361
Coward, P. D. (1990) · Elsevier eBooks · doi:10.1016/b978-0-408-03741-9.50028-3 · Cited by 1,290
Software unit test coverage and adequacy
Zhu, H. P., Hall, P. A. V., & May, J. H. R. (1997) · ACM Computing Surveys · doi:10.1145/267580.267590 · Cited by 1,252
Foundations of Machine Learning
(2012) · The MIT Press eBooks · doi:10.7551/mitpress/8291.003.0006 · Cited by 1,151
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
Introduction to Software Testing
Bierig, R., Brown, S., Galván, E., & Timoney, J. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108974073.004 · Cited by 1,056
Requirements engineering paper classification and evaluation criteria: a proposal and a discussion
Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2006) · Requirements Engineering · doi:10.1007/s00766-005-0021-6 · Cited by 940
Evidence-Based Software Engineering and Systematic Reviews
Kitchenham, B. A., Budgen, D., & Brereton, P. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/b19467 · Cited by 766
Machine Learning: The Art and Science of Algorithms that Make Sense of Data
Flach, P. (2012) · doi:10.1017/cbo9780511973000 · Cited by 753
Software testing techniques (2nd ed.)
Beizer, B. (1990) · Cited by 744
Introduction to Software Testing
Ammann, P., & Offutt, J. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316771273 · Cited by 643
Using mapping studies as the basis for further research – A participant-observer case study
Kitchenham, B. A., Budgen, D., & Brereton, O. P. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.011 · Cited by 639
Identifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 570
Harrold, M. J. (2000) · doi:10.1145/336512.336532 · Cited by 360
Guided GUI testing of android apps with minimal restart and approximate learning
Choi, W., Necula, G. C., & Sen, K. (2013) · doi:10.1145/2509136.2509552 · Cited by 320
Spieker, H., Arnaud Gotlieb, B., Marijan, D., & Mossige, M. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092709 · Cited by 290
Software testing: a research travelogue (2000–2014)
Orso, A., & Rothermel, G. (2014) · doi:10.1145/2593882.2593885 · Cited by 213
Data flow analysis techniques for test data selection
Rapps, S., & Weyuker, E. J. (2011) · Cited by 199
Λουρίδας, Π., & Ebert, C. (2016) · IEEE Software · doi:10.1109/ms.2016.114 · Cited by 190
Machine Learning and Software Engineering
Zhang, D., & Tsai, J. J. P. (2003) · Software Quality Journal · doi:10.1023/a:1023760326768 · Cited by 184
Active learning for automatic classification of software behavior
Bowring, J. F., Rehg, J. M., & Harrold, M. J. (2004) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/1007512.1007539 · Cited by 179
Zhang, J. M., Zhang, L., Harman, M., Hao, D., Jia, Y., & Zhang, L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2809496 · Cited by 137
Domain-Specific Optimization in Automata Learning
Hungar, H., Niese, O., & Steffen, B. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45069-6_31 · Cited by 117
Using a neural network in the software testing process
Vanmali, M., Last, M., & Kandel, A. (2002) · International Journal of Intelligent Systems · doi:10.1002/int.1002 · Cited by 117
Learning for test prioritization: an industrial case study
Busjaeger, B., & Xie, T. (2016) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/2950290.2983954 · Cited by 114
Ceruzzi, P. E. (2012) · The MIT Press eBooks · doi:10.7551/mitpress/9426.001.0001 · Cited by 110
Machine Learning: Hands-On for Developers and Technical Professionals
Bell, J. (2014) · Cited by 108
Kanewala, U., Bieman, J. M., & Ben‐Hur, A. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1594 · Cited by 103
The cost of errors in software development: evidence from industry
Westland, J. C. (2002) · Journal of Systems and Software · doi:10.1016/s0164-1212(01)00130-3 · Cited by 102
The role of Artificial Intelligence in Software Engineering
Harman, M. (2012) · doi:10.1109/raise.2012.6227961 · Cited by 99
Bowles, M. (2015) · Wiley eBooks · doi:10.1002/9781119183600 · Cited by 92
Using the Case-Based Ranking Methodology for Test Case Prioritization
Tonella, P., Avesani, P., & Susi, A. (2006) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2006.74 · Cited by 91
Zhang, J. M., Wang, Z., Zhang, L., Hao, D., Zang, L., Cheng, S., & Zhang, L. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931038 · Cited by 79
Novel Applications of Machine Learning in Software Testing
Briand, L. (2008) · doi:10.1109/qsic.2008.29 · Cited by 78
Test case design based on Z and the classification-tree method
Singh, H., Conrad, M., & Sadeghipour, S. (2002) · doi:10.1109/icfem.1997.630406 · Cited by 71
Bell, J. (2014) · Wiley eBooks · doi:10.1002/9781119183464 · Cited by 68
The role of artificial intelligence in software engineering
Harman, M. (2012) · Cited by 62
Using semi-supervised clustering to improve regression test selection techniques
Chen, S., Chen, Z., Zhao, Z., Xu, B., & Feng, Y. (2011) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2011.38 · Cited by 62
Testing by means of inductive program learning
Bergadano, F., & Gunetti, D. (1996) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/227607.227611 · Cited by 43
Using machine learning to refine Category-Partition test specifications and test suites
Briand, L., Labiche, Y., Bawar, Z., & Spido, N. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.06.006 · Cited by 40
Artificial Neural Network for Automatic Test Oracles Generation
Hu, J., Wang, Y., Chen, N.-W., Gou, Z.-J., & Wang, S. (2008) · doi:10.1109/csse.2008.774 · Cited by 40
An exploration of statistical models for automated test case generation
Sant, J., Souter, A. L., & Greenwald, L. G. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083256 · Cited by 39
Automatic testing of GUI‐based applications
Mariani, L., Pezzè, M., Riganelli, O., & Santoro, M. (2014) · Software Testing Verification and Reliability · doi:10.1002/stvr.1538 · Cited by 37
On the use of neural networks to guide software testing activities
Anderson, C. W., von Mayrhauser, A., & Mraz, R. (2002) · Proceedings/Proceedings - International Test Conference · doi:10.1109/test.1995.529902 · Cited by 36
Uncertainty-Driven Black-Box Test Data Generation
Walkinshaw, N., & Fraser, G. (2017) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2017.30 · Cited by 35
How high will it be? Using machine learning models to predict branch coverage in automated testing
Grano, G., Titov, T. V., Panichella, S., & Gall, H. C. (2018) · IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE) · doi:10.1109/maltesque.2018.8368454 · Cited by 32
Machine Learning Approach in Mutation Testing
Strug, J., & Strug, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34691-0_15 · Cited by 31
Linking software testing results with a machine learning approach
Lenz, A. R., Pozo, A., & Vergílio, S. R. (2013) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2013.01.008 · Cited by 31
Improving active Mealy machine learning for protocol conformance testing
Aarts, F., Kuppens, H. C., Tretmans, J., Vaandrager, F., & Verwer, S. (2013) · Machine Learning · doi:10.1007/s10994-013-5405-0 · Cited by 30
Rosenfeld, A., Kardashov, O., & Zang, O. (2018) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1145/3197231.3197241 · Cited by 30
An exploration of statistical models for automated test case generation
Sant, J., Souter, A. L., & Greenwald, L. G. (2005) · doi:10.1145/1083246.1083256 · Cited by 29
Silva, D. G., Jino, M., & de Abreu, B. T. (2010) · International Conference on Software Testing, Verification, and Validation · doi:10.1109/icst.2010.46 · Cited by 27
Assessing and generating test sets in terms of behavioural adequacy
Fraser, G., & Walkinshaw, N. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1575 · Cited by 26
Agarwal, D., Tamir, D. E., Last, M., & Kandel, A. (2012) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2012.2183590 · Cited by 25
Software testing by active learning for commercial games
Xiao, G., Southey, F., Holte, R. C., & Wilkinson, D. F. (2005) · National Conference on Artificial Intelligence · Cited by 25
Using semi-supervised learning for predicting metamorphic relations
Hardin, B., & Kanewala, U. (2018) · doi:10.1145/3193977.3193985 · Cited by 25
Browserbite: Accurate Cross-Browser Testing via Machine Learning over Image Features
Semenenko, N., Dumas, M., & Saar, T. (2013) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2013.88 · Cited by 24
Identifying Infeasible GUI Test Cases Using Support Vector Machines and Induced Grammars
Gove, R., & Faytong, J. (2011) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2011.73 · Cited by 24
Machine Learning in Value-Based Software Test Data Generation
Zhang, D. (2006) · Proceedings - International Conference on Tools with Artificial Intelligence, TAI · doi:10.1109/ictai.2006.77 · Cited by 23
An Experience-Based Approach for Test Execution Effort Estimation
Zhu, X., Zhou, B., Hou, L. X., Chen, J., & Lü, C. (2008) · International Conference for Young Computer Scientists · doi:10.1109/icycs.2008.53 · Cited by 20
Machine Learning-based Software Testing: Towards a Classification Framework.
Noorian, M., Bagheri, E., & Du, W. (2011) · Cited by 20
Predicting mutation score using source code and test suite metrics
Jalbert, K., & Bradbury, J. S. (2012) · e-scholar@UOIT (University of Ontario Institute of Technology) · doi:10.5555/2666527.2666536 · Cited by 19
Bayesian reasoning for software testing
Namin, A. S., & Sridharan, M. (2010) · doi:10.1145/1882362.1882433 · Cited by 19
Predicting mutation score using source code and test suite metrics
Jalbert, K., & Bradbury, J. S. (2012) · doi:10.1109/raise.2012.6227969 · Cited by 19
The synergy of human and artificial intelligence in software engineering
Xie, T. (2013) · doi:10.1109/raise.2013.6615197 · Cited by 18
Intelligent Test Oracle Construction for Reactive Systems without Explicit Specifications
Wang, F., Yao, L.-W., & Wu, J.-H. (2011) · IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) · doi:10.1109/dasc.2011.39 · Cited by 17
Cheatham, T. J., Yoo, J. P., & Wahl, N. J. (1995) · ACM Annual Conference on Computer Science (CSC) · doi:10.1145/259526.259548 · Cited by 17
Chan, W. K., Ho, J. C. F., & Tse, T. H. (2009) · Software Testing Verification and Reliability · doi:10.1002/stvr.408 · Cited by 17
A learning-based method for combining testing techniques
Cotroneo, D., Pietrantuono, R., & Russo, S. (2013) · doi:10.5555/2486788.2486808 · Cited by 16
A learning-based method for combining testing techniques
Cotroneo, D., Pietrantuono, R., & Russo, S. (2013) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2013.6606560 · Cited by 16
Balkan, A., Paulo Tabuada, B., Deshmukh, J. V., Jin, X., & Kapinski, J. P. (2017) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3122787 · Cited by 15
Generation of test oracles using neural network and decision tree model
Vineeta, Singhal, A., & Bansal, A. (2014) · doi:10.1109/confluence.2014.6949311 · Cited by 14
Vergílio, S. R., Maldonado, J. C., & Jino, M. (2006) · Journal of the Brazilian Computer Society · doi:10.1007/bf03192389 · Cited by 13
Testing service oriented architectures using stateful service visualization via machine learning
Enişer, H. F., & Şen, A. (2018) · International Workshop on Automation of Software Test (AST) · doi:10.1145/3194733.3194737 · Cited by 13
Revisiting AI and testing methods to infer FSM models of black-box systems
Groz, R., Simao, A. D. S., Brémond, N., & Oriat, C. (2018) · International Workshop on Automation of Software Test (AST) · doi:10.1145/3194733.3194736 · Cited by 12
Vergílio, S. R., Maldonado, J. C., & Jino, M. (2006) · Journal of the Brazilian Computer Society · doi:10.1590/s0104-65002006000200006 · Cited by 10
Artificial Intelligence and Software Engineering: Breaking the Toy Mold
Welty, C. A., & Selfridge, P. G. (1997) · Automated Software Engineering · doi:10.1023/a:1008662625094 · Cited by 9
Learning Effective Oracle Comparator Combinations for Web Applications
Sprenkle, S., Hill, E., & Pollock, L. (2007) · Proceedings of the ... International Conference on Quality Software/Proceedings · doi:10.1109/qsic.2007.4385523 · Cited by 9
Badri, M., Badri, L., Flageol, W., & Touré, F. (2017) · International Conference on Machine Learning and Soft Computing (ICMLSC) · doi:10.1145/3036290.3036323 · Cited by 7
Felbinger, H., Wotawa, F., & Nica, M. (2016) · International Workshop on Automation of Software Test (AST) · doi:10.1145/2896921.2896923 · Cited by 6
Research works printed but not linked: 11
- choi, 0, Guided GUI testing of android apps with minimal restart and approximate learning, Proc ACM SIGPLAN Int Conf Object Oriented Program Syst Lang Appl, 623
- noorian, 0, Machine learning-based software testing: Towards a classification framework, Proc Int Conf on Software Eng Knowledge Eng, 225
- 10.1109/TEST.1995.529902 · DOI
- 10.1109/QSIC.2008.29 · DOI
- 10.1109/ICFEM.1997.630406 · DOI
- stone, 2012, Bayes’ Rule A Tutorial Introduction to Bayesian Analysis
- lantz, 2015, Machine Learning with R
- 10.1145/1007512.1007539 · DOI
- myers, 2011, The Art of Software Testing
- 10.1145/2896921.2896923 · DOI
- stringer, 2013, Action Research
Cited by
Sorted by citations
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Test case selection and prioritization using machine learning: a systematic literature review
Pan, R., Bagherzadeh, M., Ghaleb, T. A., & Briand, L. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10066-6 · Cited by 220
Reinforcement Learning for Test Case Prioritization
Bagherzadeh, M., Kahani, N., & Briand, L. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3070549 · Cited by 157
AI-driven innovations in software engineering: a review of current practices and future directions
Alenezi, M., & Akour, M. (2025) · Applied Sciences · doi:10.3390/app15031344 · Cited by 121
Metadata
- Authors
- Durelli, V. H. S., Durelli, R. S., Borges, S. S., Endo, A. T., Eler, M. M., Dias, D. R. C., & Guimaraes, M. P.
- Affiliations
- Universidade Federal de Lavras (BR)
- Universidade Tecnológica Federal do Paraná (BR)
- Universidade de São Paulo (BR)
- Federal University of São João del-Rei (BR)
- ORCID
- Venue
- IEEE Transactions on Reliability
- Published
- 2019-09
- Volume
- 68
- Issue
- 3
- Pages
- 1189-1212
- ISSN
- 0018-9529 (ISSN-L)
- 1558-1721
- OpenAlex
- W2912899164
- Cited by
- 207
- References
- 93
- Citations by year
- 2026: 13 · 2025: 42 · 2024: 33 · 2023: 28 · 2022: 36 · 2021: 23 · 2020: 21 · 2019: 5
- FWCI
- 51.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Reliability (publishedVersion)
- Funders
- CNPq/Brazil (420363/2018-1)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- machine learning, software testing, test automation, test case generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
