Machine learning applied to software testing: A systematic mapping study
Abstract
Software testing involves probing into the behavior of software systems to uncover faults. Most testing activities are complex and costly, so a practical strategy that has been adopted to circumvent these issues is to automate software testing. There has been a growing interest in applying machine learning (ML) to automate various software engineering activities, including testing-related ones. In this paper, we set out to review the state-of-the art of how ML has been explored to automate and streamline software testing and provide an overview of the research at the intersection of these two fields by conducting a systematic mapping study. We selected 48 primary studies. These selected studies were then categorized according to study type, testing activity, and ML algorithm employed to automate the testing activity. The results highlight the most widely used ML algorithms and identify several avenues for future research. We found that ML algorithms have been used mainly for test-case generation, refinement, and evaluation. Also, ML has been used to evaluate test oracle construction and to predict the cost of testing-related activities. The results of this paper outline the ML algorithms that are most commonly used to automate software-testing activities, helping researchers to understand the current state of research concerning ML applied to software testing. We also found that there is a need for better empirical studies examining how ML algorithms have been used to automate software-testing activities.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Test case evaluation is a complex task since it is hard to formalize and measure test cases' characteristics influence quality.
“Test case evaluation is a complex task since it is hard to formalize and measure test cases' characteristics influence quality (Durelli et al., 2019).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Durelli et al., 2019
Cites
(1963) · Medical Entomology and Zoology · Cited by 15,085 · see citing works
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106 · see citing works
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460 · see citing works
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 2× · see citing works
“Throughout the backward snowballing process, we followed the guidelines provided by Wohlin [35].”
BackgroundMethodUnderstanding Machine Learning: From Theory to Algorithms
Shalev-Shwartz, S., & Ben-David, S. (2014) · doi:10.1017/cbo9781107298019 · Cited by 3,115 · see citing works
(2015) · Cited by 2,857 · see citing works
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669 · see citing works
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,750 · see citing works
The Goal Question Metric Approach
Basili, V. R., Caldiera, G., & Rombach, H. D. (1994) · Cited by 1,587 · see citing works
Prioritizing test cases for regression testing
Rothermel, G., Untch, R. H., Chu, C., & Harrold, M. J. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.962562 · Cited by 1,361 · see citing works
Coward, P. D. (1990) · Elsevier eBooks · doi:10.1016/b978-0-408-03741-9.50028-3 · Cited by 1,290 · see citing works
Software unit test coverage and adequacy
Zhu, H. P., Hall, P. A. V., & May, J. H. R. (1997) · ACM Computing Surveys · doi:10.1145/267580.267590 · Cited by 1,252 · see citing works
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104 · see citing works
Foundations of Machine Learning
(2012) · The MIT Press eBooks · doi:10.7551/mitpress/8291.003.0006 · Cited by 1,102 · see citing works
Introduction to Software Testing
Bierig, R., Brown, S., Galván, E., & Timoney, J. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108974073.004 · Cited by 1,056 · see citing works
Requirements engineering paper classification and evaluation criteria: a proposal and a discussion
Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2006) · Requirements Engineering · doi:10.1007/s00766-005-0021-6 · Cited by 940 · see citing works
Evidence-Based Software Engineering and Systematic Reviews
Kitchenham, B. A., Budgen, D., & Brereton, P. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/b19467 · Cited by 766 · see citing works
Machine Learning: The Art and Science of Algorithms that Make Sense of Data
Flach, P. (2012) · doi:10.1017/cbo9780511973000 · Cited by 753 · see citing works
Software testing techniques (2nd ed.)
Beizer, B. (1990) · Cited by 744 · see citing works
Using mapping studies as the basis for further research – A participant-observer case study
Kitchenham, B. A., Budgen, D., & Brereton, O. P. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.011 · Cited by 639 · see citing works
Introduction to Software Testing
Ammann, P., & Offutt, J. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316771273 · Cited by 609 · see citing works
Identifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 570 · see citing works
Harrold, M. J. (2000) · doi:10.1145/336512.336532 · Cited by 360 · see citing works
Guided GUI testing of android apps with minimal restart and approximate learning
Choi, W., Necula, G. C., & Sen, K. (2013) · doi:10.1145/2509136.2509552 · Cited by 320 · see citing works
Spieker, H., Arnaud Gotlieb, B., Marijan, D., & Mossige, M. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092709 · Cited by 290 · see citing works
Software testing: a research travelogue (2000–2014)
Orso, A., & Rothermel, G. (2014) · doi:10.1145/2593882.2593885 · Cited by 213 · see citing works
Data flow analysis techniques for test data selection
Rapps, S., & Weyuker, E. J. (2011) · Cited by 199 · see citing works
Λουρίδας, Π., & Ebert, C. (2016) · IEEE Software · doi:10.1109/ms.2016.114 · Cited by 190 · see citing works
Machine Learning and Software Engineering
Zhang, D., & Tsai, J. J. P. (2003) · Software Quality Journal · doi:10.1023/a:1023760326768 · Cited by 184 · see citing works
Active learning for automatic classification of software behavior
Bowring, J. F., Rehg, J. M., & Harrold, M. J. (2004) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/1007512.1007539 · Cited by 179 · see citing works
Zhang, J. M., Zhang, L., Harman, M., Hao, D., Jia, Y., & Zhang, L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2809496 · Cited by 137 · see citing works
Domain-Specific Optimization in Automata Learning
Hungar, H., Niese, O., & Steffen, B. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45069-6_31 · Cited by 117 · see citing works
Using a neural network in the software testing process
Vanmali, M., Last, M., & Kandel, A. (2002) · International Journal of Intelligent Systems · doi:10.1002/int.1002 · Cited by 117 · see citing works
Learning for test prioritization: an industrial case study
Busjaeger, B., & Xie, T. (2016) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/2950290.2983954 · Cited by 114 · see citing works
Ceruzzi, P. E. (2012) · The MIT Press eBooks · doi:10.7551/mitpress/9426.001.0001 · Cited by 110 · see citing works
Machine Learning: Hands-On for Developers and Technical Professionals
Bell, J. (2014) · Cited by 108 · see citing works
Kanewala, U., Bieman, J. M., & Ben‐Hur, A. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1594 · Cited by 103 · see citing works
The cost of errors in software development: evidence from industry
Westland, J. C. (2002) · Journal of Systems and Software · doi:10.1016/s0164-1212(01)00130-3 · Cited by 102 · see citing works
The role of Artificial Intelligence in Software Engineering
Harman, M. (2012) · doi:10.1109/raise.2012.6227961 · Cited by 99 · see citing works
Bowles, M. (2015) · Wiley eBooks · doi:10.1002/9781119183600 · Cited by 92 · see citing works
Using the Case-Based Ranking Methodology for Test Case Prioritization
Tonella, P., Avesani, P., & Susi, A. (2006) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2006.74 · Cited by 91 · see citing works
Zhang, J. M., Wang, Z., Zhang, L., Hao, D., Zang, L., Cheng, S., & Zhang, L. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931038 · Cited by 79 · see citing works
Novel Applications of Machine Learning in Software Testing
Briand, L. (2008) · doi:10.1109/qsic.2008.29 · Cited by 78 · see citing works
Test case design based on Z and the classification-tree method
Singh, H., Conrad, M., & Sadeghipour, S. (2002) · doi:10.1109/icfem.1997.630406 · Cited by 71 · see citing works
Bell, J. (2014) · Wiley eBooks · doi:10.1002/9781119183464 · Cited by 68 · see citing works
The role of artificial intelligence in software engineering
Harman, M. (2012) · Cited by 62 · see citing works
Using semi-supervised clustering to improve regression test selection techniques
Chen, S., Chen, Z., Zhao, Z., Xu, B., & Feng, Y. (2011) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2011.38 · Cited by 62 · see citing works
Testing by means of inductive program learning
Bergadano, F., & Gunetti, D. (1996) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/227607.227611 · Cited by 43 · see citing works
Using machine learning to refine Category-Partition test specifications and test suites
Briand, L., Labiche, Y., Bawar, Z., & Spido, N. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.06.006 · Cited by 40 · see citing works
Artificial Neural Network for Automatic Test Oracles Generation
Hu, J., Wang, Y., Chen, N.-W., Gou, Z.-J., & Wang, S. (2008) · doi:10.1109/csse.2008.774 · Cited by 40 · see citing works
An exploration of statistical models for automated test case generation
Sant, J., Souter, A. L., & Greenwald, L. G. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083256 · Cited by 39 · see citing works
Automatic testing of GUI‐based applications
Mariani, L., Pezzè, M., Riganelli, O., & Santoro, M. (2014) · Software Testing Verification and Reliability · doi:10.1002/stvr.1538 · Cited by 37 · see citing works
On the use of neural networks to guide software testing activities
Anderson, C. W., von Mayrhauser, A., & Mraz, R. (2002) · Proceedings/Proceedings - International Test Conference · doi:10.1109/test.1995.529902 · Cited by 36 · see citing works
Uncertainty-Driven Black-Box Test Data Generation
Walkinshaw, N., & Fraser, G. (2017) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2017.30 · Cited by 35 · see citing works
Machine Learning Approach in Mutation Testing
Strug, J., & Strug, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34691-0_15 · Cited by 32 · see citing works
How high will it be? Using machine learning models to predict branch coverage in automated testing
Grano, G., Titov, T. V., Panichella, S., & Gall, H. C. (2018) · IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE) · doi:10.1109/maltesque.2018.8368454 · Cited by 32 · see citing works
Linking software testing results with a machine learning approach
Lenz, A. R., Pozo, A., & Vergílio, S. R. (2013) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2013.01.008 · Cited by 31 · see citing works
Improving active Mealy machine learning for protocol conformance testing
Aarts, F., Kuppens, H. C., Tretmans, J., Vaandrager, F., & Verwer, S. (2013) · Machine Learning · doi:10.1007/s10994-013-5405-0 · Cited by 30 · see citing works
Rosenfeld, A., Kardashov, O., & Zang, O. (2018) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1145/3197231.3197241 · Cited by 30 · see citing works
An exploration of statistical models for automated test case generation
Sant, J., Souter, A. L., & Greenwald, L. G. (2005) · doi:10.1145/1083246.1083256 · Cited by 29 · see citing works
Silva, D. G., Jino, M., & de Abreu, B. T. (2010) · International Conference on Software Testing, Verification, and Validation · doi:10.1109/icst.2010.46 · Cited by 27 · see citing works
Assessing and generating test sets in terms of behavioural adequacy
Fraser, G., & Walkinshaw, N. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1575 · Cited by 26 · see citing works
Agarwal, D., Tamir, D. E., Last, M., & Kandel, A. (2012) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2012.2183590 · Cited by 25 · see citing works
Software testing by active learning for commercial games
Xiao, G., Southey, F., Holte, R. C., & Wilkinson, D. F. (2005) · National Conference on Artificial Intelligence · Cited by 25 · see citing works
Using semi-supervised learning for predicting metamorphic relations
Hardin, B., & Kanewala, U. (2018) · doi:10.1145/3193977.3193985 · Cited by 25 · see citing works
Browserbite: Accurate Cross-Browser Testing via Machine Learning over Image Features
Semenenko, N., Dumas, M., & Saar, T. (2013) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2013.88 · Cited by 24 · see citing works
Identifying Infeasible GUI Test Cases Using Support Vector Machines and Induced Grammars
Gove, R., & Faytong, J. (2011) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2011.73 · Cited by 24 · see citing works
Machine Learning in Value-Based Software Test Data Generation
Zhang, D. (2006) · Proceedings - International Conference on Tools with Artificial Intelligence, TAI · doi:10.1109/ictai.2006.77 · Cited by 23 · see citing works
An Experience-Based Approach for Test Execution Effort Estimation
Zhu, X., Zhou, B., Hou, L. X., Chen, J., & Lü, C. (2008) · International Conference for Young Computer Scientists · doi:10.1109/icycs.2008.53 · Cited by 20 · see citing works
Machine Learning-based Software Testing: Towards a Classification Framework.
Noorian, M., Bagheri, E., & Du, W. (2011) · Cited by 20 · see citing works
Predicting mutation score using source code and test suite metrics
Jalbert, K., & Bradbury, J. S. (2012) · e-scholar@UOIT (University of Ontario Institute of Technology) · doi:10.5555/2666527.2666536 · Cited by 19 · see citing works
Bayesian reasoning for software testing
Namin, A. S., & Sridharan, M. (2010) · doi:10.1145/1882362.1882433 · Cited by 19 · see citing works
Predicting mutation score using source code and test suite metrics
Jalbert, K., & Bradbury, J. S. (2012) · doi:10.1109/raise.2012.6227969 · Cited by 19 · see citing works
The synergy of human and artificial intelligence in software engineering
Xie, T. (2013) · doi:10.1109/raise.2013.6615197 · Cited by 18 · see citing works
Cheatham, T. J., Yoo, J. P., & Wahl, N. J. (1995) · ACM Annual Conference on Computer Science (CSC) · doi:10.1145/259526.259548 · Cited by 17 · see citing works
Chan, W. K., Ho, J. C. F., & Tse, T. H. (2009) · Software Testing Verification and Reliability · doi:10.1002/stvr.408 · Cited by 17 · see citing works
Intelligent Test Oracle Construction for Reactive Systems without Explicit Specifications
Wang, F., Yao, L.-W., & Wu, J.-H. (2011) · IEEE International Conference on Dependable, Autonomic and Secure Computing · doi:10.1109/dasc.2011.39 · Cited by 16 · see citing works
A learning-based method for combining testing techniques
Cotroneo, D., Pietrantuono, R., & Russo, S. (2013) · doi:10.5555/2486788.2486808 · Cited by 16 · see citing works
A learning-based method for combining testing techniques
Cotroneo, D., Pietrantuono, R., & Russo, S. (2013) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2013.6606560 · Cited by 16 · see citing works
Generation of test oracles using neural network and decision tree model
Vineeta, Singhal, A., & Bansal, A. (2014) · doi:10.1109/confluence.2014.6949311 · Cited by 14 · see citing works
Vergílio, S. R., Maldonado, J. C., & Jino, M. (2006) · Journal of the Brazilian Computer Society · doi:10.1007/bf03192389 · Cited by 13 · see citing works
Testing service oriented architectures using stateful service visualization via machine learning
Enişer, H. F., & Şen, A. (2018) · International Workshop on Automation of Software Test (AST) · doi:10.1145/3194733.3194737 · Cited by 13 · see citing works
Balkan, A., Tabuada, P., Deshmukh, J. V., Jin, X., & Kapinski, J. P. (2017) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3122787 · Cited by 12 · see citing works
Revisiting AI and testing methods to infer FSM models of black-box systems
Groz, R., Simao, A. D. S., Brémond, N., & Oriat, C. (2018) · International Workshop on Automation of Software Test (AST) · doi:10.1145/3194733.3194736 · Cited by 12 · see citing works
Vergílio, S. R., Maldonado, J. C., & Jino, M. (2006) · Journal of the Brazilian Computer Society · doi:10.1590/s0104-65002006000200006 · Cited by 10 · see citing works
Artificial Intelligence and Software Engineering: Breaking the Toy Mold
Welty, C. A., & Selfridge, P. G. (1997) · Automated Software Engineering · doi:10.1023/a:1008662625094 · Cited by 9 · see citing works
Learning Effective Oracle Comparator Combinations for Web Applications
Sprenkle, S., Hill, E., & Pollock, L. (2007) · Proceedings of the ... International Conference on Quality Software/Proceedings · doi:10.1109/qsic.2007.4385523 · Cited by 9 · see citing works
Badri, M., Badri, L., Flageol, W., & Touré, F. (2017) · International Conference on Machine Learning and Soft Computing (ICMLSC) · doi:10.1145/3036290.3036323 · Cited by 7 · see citing works
Felbinger, H., Wotawa, F., & Nica, M. (2016) · International Workshop on Automation of Software Test (AST) · doi:10.1145/2896921.2896923 · Cited by 6 · see citing works
Flagged first, then by citations
4 of the 93 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241 · see citing works
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
Test case selection and prioritization using machine learning: a systematic literature review
Pan, R., Bagherzadeh, M., Ghaleb, T. A., & Briand, L. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10066-6 · Cited by 197 · see citing works
Reinforcement Learning for Test Case Prioritization
Bagherzadeh, M., Kahani, N., & Briand, L. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3070549 · Cited by 147 · see citing works
AI-driven innovations in software engineering: a review of current practices and future directions
Alenezi, M., & Akour, M. (2025) · Applied Sciences · doi:10.3390/app15031344 · Cited by 121 · see citing works
Engineering AI Systems: A Research Agenda
Bosch, J., Olsson, H. H., & Crnkovic, I. (2021) · Advances in Systems Analysis, Software Engineering, and High Performance Computing · doi:10.4018/978-1-7998-5101-1.ch001 · Cited by 108 · see citing works
Deep learning-based software engineering: progress, challenges, and opportunities
Chen, X., Hu, X., Huang, Y., Jiang, H., Ji, W., Jiang, Y., Jiang, Y., Liu, B., Liu, H., Li, X., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-4127-5 · Cited by 114 · see citing works
When Software Security Meets Large Language Models: A Survey
Zhu, X., Zhou, W., Han, Q., Ma, W., Wen, S., & Xiang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124971 · Cited by 105 · see citing works
Batool, I., & Khan, T. A. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.107886 · Cited by 96 · see citing works
Learning-to-rank vs ranking-to-learn
Bertolino, A., Guerriero, A., Miranda, B., Pietrantuono, R., & Russo, S. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380369 · Cited by 86 · see citing works
FlakeFlagger: Predicting Flakiness Without Rerunning Tests
Alshammari, A., Morris, C., Hilton, M. L., & Bell, J. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00140 · Cited by 75 · see citing works
A Literature Review of Using Machine Learning in Software Development Life Cycle Stages
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2021) · IEEE Access · doi:10.1109/access.2021.3119746 · Cited by 79 · see citing works
An Overview of Adaptive-Surrogate-Model-Assisted Methods for Reliability-Based Design Optimization
Ling, C., Kuo, W., & Xie, M. (2022) · IEEE Transactions on Reliability · doi:10.1109/tr.2022.3200137 · Cited by 68 · see citing works
A Survey on Machine Learning Techniques for Source Code AnalysisPreprint
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R. K., Vats, I., Moazen, H., & Sarro, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09610 · Cited by 56 · see citing works
Systematic Mapping: Artificial Intelligence Techniques in Software Engineering
Sofian, H., Yunus, N. A. M., & Ahmad, R. B. (2022) · IEEE Access · doi:10.1109/access.2022.3174115 · Cited by 54 · see citing works
A systematic review of machine learning methods in software testing
Ajorloo, S., Jamarani, A., Kashfi, M., Kashani, M. H., & Najafizadeh, A. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111805 · Cited by 53 · see citing works
Artificial Intelligence Applied to Software Testing: A Tertiary Study
Amalfitano, D., Faralli, S., Hauck, J. C. R., Matalonga, S., & Distante, D. (2023) · ACM Computing Surveys · doi:10.1145/3616372 · Cited by 47 · see citing works
Artificial Intelligence in Software Testing: A Systematic Review
Islam, M., Khan, F., Alam, S., & Hasan, M. (2023) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon58879.2023.10322349 · Cited by 46 · see citing works
A survey on machine learning techniques applied to source code
Sharma, T., Kechagia, M., Georgiou, S., Tiwari, R., Vats, I., Moazen, H., & Sarro, F. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111934 · Cited by 130 · see citing works
Machine learning in software defect prediction: A business-driven systematic mapping study
Stradowski, S., & Madeyski, L. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107128 · Cited by 41 · see citing works
Machine Learning for Software Engineering: A Tertiary Study
Kotti, Z., Galanopoulou, R., & Spinellis, D. D. (2022) · ACM Computing Surveys · doi:10.1145/3572905 · Cited by 44 · see citing works
Stradowski, S., & Madeyski, L. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107192 · Cited by 34 · see citing works
Towards automated video game testing
Politowski, C., Guéhéneuc, Y., & Petrillo, F. D. S. (2022) · doi:10.1145/3524494.3527627 · Cited by 30 · see citing works
Test Suite Optimization Using Machine Learning Techniques: A Comprehensive Study
Mehmood, A., Ilyas, Q. M., Ahmad, M., & Shi, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3490453 · Cited by 28 · see citing works
Machine Learning in Requirements Engineering: A Mapping Study
Zamani, K., Zowghi, D., & Arora, C. (2021) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew53955.2021.00023 · Cited by 28 · see citing works
Root causing, detecting, and fixing flaky tests: State of the art and future roadmap
Zolfaghari, B., Parizi, R. M., Srivastava, G., & Hailemariam, Y. (2020) · Software Practice and Experience · doi:10.1002/spe.2929 · Cited by 27 · see citing works
Assessing test artifact quality—A tertiary study
Tran, H. K. V., Unterkalmsteiner, M., Börstler, J., & bin Ali, N. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106620 · Cited by 26 · see citing works
A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks
Casey, B., Santos, J. C. S., & Perry, G. (2025) · ACM Computing Surveys · doi:10.1145/3721977 · Cited by 28 · see citing works
An Improved Crow Search Algorithm for Test Data Generation Using Search-Based Mutation Testing
Jatana, N., & Suri, B. (2020) · Neural Processing Letters · doi:10.1007/s11063-020-10288-7 · Cited by 25 · see citing works
Arrieta, A., Ayerdi, J., Illarramendi, M., Agirre, A., Sagardui, G., & Arratibel, M. (2021) · doi:10.1109/ast52587.2021.00012 · Cited by 25 · see citing works
The integration of machine learning into automated test generation: A systematic mapping study
Fontes, A., & Gay, G. (2023) · Software Testing Verification and Reliability · doi:10.1002/stvr.1845 · Cited by 24 · see citing works
Trend Application of Machine Learning in Test Case Prioritization: A Review on Techniques
Khatibsyarbini, M., Isa, M. A., Jawawi, D. N. A., Mohd-Shafie, M. L., Kadir, W. M. N. W., Hamed, H. N. A., & Suffian, M. D. M. (2021) · IEEE Access · doi:10.1109/access.2021.3135508 · Cited by 22 · see citing works
A Review of AI-augmented End-to-End Test Automation Tools
Pham, P., Nguyen, V., & Nguyen, T. N. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3563240 · Cited by 21 · see citing works
Chapter 1 Climbing the Stairway to Heaven
Olsson, H. H., Alahyari, H., & Bosch, J. (2012) · Springer eBooks · doi:10.1007/978-3-031-10873-0_2 · Cited by 21 · see citing works
Systematic Literature Review on Test Case Selection and Prioritization: A Tertiary Study
Singhal, S., Jatana, N., Suri, B., Misra, S., & Sanz, L. F. (2021) · Applied Sciences · doi:10.3390/app112412121 · Cited by 20 · see citing works
Brito, C., Durelli, V. H. S., Durelli, R. S., Souza, S. R. S., Vincenzi, A. M. R., & Delamaro, M. E. (2020) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw50294.2020.00056 · Cited by 19 · see citing works
Self-citationMadeyski, L., & Stradowski, S. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112360 · Cited by 18 · see citing works
Kotenko, I. V., Евгеньевич, И. К., Buinevich, M., Saenko, I. B., & Shorey, R. (2023) · Energies · doi:10.3390/en16135111 · Cited by 15 · see citing works
Systematic Literature Review on Application of Learning-Based Approaches in Continuous Integration
Arani, A. K., Le, T. H. M., Zahedi, M., & Babar, M. A. (2024) · IEEE Access · doi:10.1109/access.2024.3424276 · Cited by 15 · see citing works
A multi-year grey literature review on AI-assisted test automation
Ricca, F., Marchetto, A., & Stocco, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107799 · Cited by 15 · see citing works
Gartziandia, A., Arrieta, A., Ayerdi, J., Illarramendi, M., Agirre, A., Sagardui, G., & Arratibel, M. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2465 · Cited by 14 · see citing works
Using deep learning for selenium web UI functional tests: A case-study with e-commerce applications
Khaliq, Z., Khan, D. A., & Farooq, S. U. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105446 · Cited by 14 · see citing works
A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning
Ramírez, A., FELDT, R. H., & Romero, J. R. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3511805 · Cited by 13 · see citing works
Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey
She, X., Liu, Y., Zhao, Y., He, Y., Li, L., Tantithamthavorn, C. K., Qin, Z., & Wang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748647 · Cited by 23 · see citing works
Rajagopal, M., Ramkumar, S., Loganathan, K., & Sarris, I. E. (2023) · Information · doi:10.3390/info14030166 · Cited by 13 · see citing works
Using machine learning to generate test oracles: a systematic literature review
Fontes, A., & Gay, G. (2021) · doi:10.1145/3472675.3473974 · Cited by 13 · see citing works
Panda, P., Sahoo, D., & Sahoo, D. (2024) · International Conference on Sustainable Computing and Smart Systems (ICSCSS) · doi:10.1109/icscss60660.2024.10625524 · Cited by 11 · see citing works
AutoTestGPT: A System for the Automated Generation of Software Test Cases based on ChatGPT
Liu, H., Liu, L., Yue, C., Wang, Y., & Deng, B. (2024) · Journal of Software · doi:10.17706/jsw.19.4.108-120 · Cited by 11 · see citing works
A comprehensive review for test case prioritization in Software Engineering
Ashima, A., Shaheamlung, G., & Rote, K. (2020) · International Conference on Intelligent Engineering and Management (ICIEM) · doi:10.1109/iciem48762.2020.9160217 · Cited by 11 · see citing works
VALIDATE: A deep dive into vulnerability prediction datasets
Esposito, M., & Falessi, D. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107448 · Cited by 10 · see citing works
Potential Bottleneck and Measuring Performance of Serverless Computing: A Literature Study
Khatri, D., Khatri, S. K., & Mishra, D. (2020) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito48877.2020.9197837 · Cited by 10 · see citing works
Test Automation Framework using Soft Computing Techniques
Swathi, B., & Tiwari, H. (2021) · International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) · doi:10.1109/icaect49130.2021.9392602 · Cited by 10 · see citing works
On Using Decision Tree Coverage Criteria forTesting Machine Learning Models
Santos, S. H. N., Silveira, B., Durelli, V. H. S., Durelli, R. S., Souza, S. R. S., & Delamaro, M. E. (2021) · Brazilian Symposium on Systematic and Automated Software Testing (SAST) · doi:10.1145/3482909.3482911 · Cited by 9 · see citing works
Self-citationReanzi, S. R. D., & Thangaiah, P. R. J. (2021) · International Journal of Engineering Business Management · doi:10.1177/18479790211062044 · Cited by 9 · see citing works
Machine Learning for Software Engineering: A Systematic MappingPreprint
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.13299 · Cited by 11 · see citing works
SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems
Yang, H., Zhou, Y., & Chen, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3744242 · Cited by 8 · see citing works
TestLab: An Intelligent Automated Software Testing Framework
Dias, T., Batista, A., Maia, E., & Praça, I. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-38318-2_35 · Cited by 8 · see citing works
Predicting Test Case Verdicts Using Textual Analysis of Committed Code Churns.
Al-Sabbagh, K. W., Staron, M., Hebig, R., & Meding, W. (2019) · Cited by 8 · see citing works
Neural Network Embeddings for Test Case PrioritizationPreprint
Lousada, J., & Ribeiro, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.10154 · Cited by 8 · see citing works
Sustainability Integration of Artificial Intelligence into the Software Development Life Cycle
Trinh, E., Funke, M., Lago, P., & Bogner, J. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00044 · Cited by 7 · see citing works
Qian, Z., Yu, Q., Hui, Z., Liu, J., & Fu, T. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107654 · Cited by 7 · see citing works
Tanzil, M. H., Chowdhury, S., Modaberi, S., Uddin, G., & Hemmati, H. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112405 · Cited by 7 · see citing works
Artificial Intelligence For Software Testing-Perspectives And Practices
Jha, N., & Popli, R. (2021) · International Conference on Computational Intelligence and Communication Technologies (CCICT) · doi:10.1109/ccict53244.2021.00075 · Cited by 7 · see citing works
Exploring potential implications of intelligent tools for human aspects of software engineering
Melegati, J., Nascimento, N., Chanin, R. M., Sales, A., & Wiese, I. (2024) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3641822.3641877 · Cited by 7 · see citing works
Understanding the factors that influence software testing through moments of translation
Sekgweleo, T., & Iyamu, T. (2022) · Journal of Systems and Information Technology · doi:10.1108/jsit-07-2021-0125 · Cited by 7 · see citing works
Using Machine Learning on Testing IoT Applications: a systematic mapping
Freitas, L. M., & Lelli, V. (2022) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3539637.3558049 · Cited by 7 · see citing works
Chapter 13 Engineering AI Systems
Bosch, J., Olsson, H. H., & Crnković, I. (2021) · Springer eBooks · doi:10.1007/978-3-031-10873-0_18 · Cited by 6 · see citing works
Reasoning-Based Software Testing
Giamattei, L., Pietrantuono, R., & Russo, S. (2023) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier58687.2023.00018 · Cited by 6 · see citing works
Kang, H., & Do, S. (2024) · Electronics · doi:10.3390/electronics13091690 · Cited by 6 · see citing works
Agent-based Testing of Extended Reality Systems
Prada, R., Prasetya, I. S. W. B., Kifetew, F. M., Dignum, F. P. M., Vos, T. E. J., Lander, J., Donnart, J.-Y., Kazmierowski, A., Davidson, J., & Fernandes, P. M. (2020) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst46399.2020.00051 · Cited by 6 · see citing works
High-throughput design and performance validation of superior latent heat eutectic salt materials
Yang, F., Xuan, Y., & Liu, X. (2025) · Journal of Energy Storage · doi:10.1016/j.est.2025.115864 · Cited by 5 · see citing works
AutoQALLMs: Automating Web Application Testing Using Large Language Models (LLMs) and Selenium
Mallipeddi, S., Yaqoob, M., Khan, J. A., Mehmood, T., Mylonas, A., & Pitropakis, N. (2025) · Computers · doi:10.3390/computers14110501 · Cited by 5 · see citing works
A Novel Vision-Based Approach to Test Sequence Generation for Mobile GUI Testing
Cui, C., Huang, Y., Huang, R., Zhou, L., Wang, R., & Li, T. (2026) · IEEE Transactions on Reliability · doi:10.1109/tr.2025.3648728 · Cited by 5 · see citing works
A Novel approach of GUI Mapping with image based widget detection and classification
Jaganeshwari, K., & Djodilatchoumy, S. (2021) · International Conference on Intelligent Engineering and Management (ICIEM) · doi:10.1109/iciem51511.2021.9445281 · Cited by 5 · see citing works
Automatically Tagging the “AAA” Pattern in Unit Test Cases Using Machine Learning Models
Wei, C., Xiao, L., Yu, T., Chen, X., Wang, X., Wong, S., & Clune, A. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3252442 · Cited by 5 · see citing works
Stradowski, S., & Madeyski, L. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00026 · Cited by 5 · see citing works
Yue, S. (2024) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-024-00591-0 · Cited by 7 · see citing works
Shao, Y., Liu, B., Wang, S., & Xiao, P. (2020) · Eksploatacja i Niezawodnosc - Maintenance and Reliability · doi:10.17531/ein.2020.3.4 · Cited by 5 · see citing works
Robotic Automation of Software Testing From a Machine Learning Viewpoint
Yadav, V., Botchway, R. K., Šenkeřík, R., & Oplatková, Z. K. (2021) · MENDEL · doi:10.13164/mendel.2021.2.068 · Cited by 5 · see citing works
Test Data Selection Based on Applying Mutation Testing to Decision Tree Models
Silveira, B., Durelli, V. H. S., Santos, S. H. N., Durelli, R. S., Delamaro, M. E., & Souza, S. R. S. (2023) · Brazilian Symposium on Systematic and Automated Software Testing (SAST) · doi:10.1145/3624032.3624038 · Cited by 5 · see citing works
Self-citationA Two-Stage Framework for Ambiguous Classification in Software Engineering
Li, J., Lei, Y., Li, S., Zhou, H., Yu, Y., Jia, Z., Ma, Y., & Wang, T. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00070 · Cited by 6 · see citing works
The qualitative factor in software testing: A systematic mapping study of qualitative methods
Ardic, B., Brandt, C., Khatami, A., Swillus, M., & Zaidman, A. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112447 · Cited by 10 · see citing works
Mohapatra, H., Pramanik, S., & Mishra, S. R. (2025) · Artificial intelligence-enhanced software and systems engineering · doi:10.1007/978-3-031-88188-6_3 · Cited by 4 · see citing works
Refining software defect prediction through attentive neural models for code understanding
Nashaat, M., & Miller, J. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112266 · Cited by 5 · see citing works
Joy, M. S., Alam, G. T., & Bakhsh, M. M. (2024) · Journal of Artificial Intelligence General science (JAIGS) · doi:10.60087/jaigs.v7i01.297 · Cited by 4 · see citing works
Regression Testing in Agile—A Systematic Mapping Study
Das, S., & Gary, K. A. (2025) · Software · doi:10.3390/software4020009 · Cited by 4 · see citing works
Application of Machine Learning on Software Quality Assurance and Testing: A Chronological Survey
Chen, H., & Hossain, M. A. (2022) · EPiC series in computing · doi:10.29007/5p9l · Cited by 4 · see citing works
Metamorphic relation automation: Rationale, challenges, and solution directions
Altamimi, E., Elkawakjy, A., & Catal, C. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2509 · Cited by 4 · see citing works
Moghaddam, M. M., Aghaei, A. A., & Parand, K. (2024) · International Conference on Distributed Computing and High Performance Computing (DCHPC) · doi:10.1109/dchpc60845.2024.10454075 · Cited by 4 · see citing works
Reimagining Unit Test Generation With AI: A Journey From Evolutionary Models to Transformers
Esubalew, S. Z., & Assefa, B. G. (2025) · IEEE Access · doi:10.1109/access.2025.3597049 · Cited by 3 · see citing works
Mudassir, G., & Marco, A. D. (2025) · Sustainability · doi:10.3390/su172210035 · Cited by 3 · see citing works
Ochodek, M., Hebig, R., Meding, W., Frost, G., & Staron, M. (2019) · Springer eBooks · doi:10.1007/978-3-031-10873-0_11 · Cited by 3 · see citing works
When debugging encounters artificial intelligence: state of the art and open challenges
Song, Y., Xie, X., & Xu, B. (2024) · Science China Information Sciences · doi:10.1007/s11432-022-3803-9 · Cited by 3 · see citing works
Software Quality Assessment Model: A New Approach for Software Testing Tools
Zulkifli, Z., Mardiana, M., & Despa, D. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194024500517 · Cited by 3 · see citing works
Can explainable artificial intelligence support software modelers in model comprehension?
Alcaide, F. J., Romero, J. R., & Ramírez, A. (2025) · Software & Systems Modeling · doi:10.1007/s10270-024-01251-4 · Cited by 3 · see citing works
Software Test Automation Using Selenium and Machine Learning
Jha, N., Popli, R., Chakraborty, S., & Kumar, P. R. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-16-6246-1_35 · Cited by 3 · see citing works
Enhancing Software Testing with Machine Learning Techniques
Akila, V., Vasuki, A., Christaline, J. A., Sathiya, R., Rishi, P., & Edward, A. S. (2023) · International Conference on Sustainable Computing and Data Communication Systems (ICSCDS) · doi:10.1109/icscds56580.2023.10105028 · Cited by 3 · see citing works
Review of Software Model-Checking Techniques for Dealing with Error Detection in Program Codes
Aliyu, E. O. (2023) · Journal of Software Engineering and Applications · doi:10.4236/jsea.2023.166010 · Cited by 3 · see citing works
Towards the definition of a research agenda on mobile application testing based on a tertiary study
Kuroishi, P. H., Maldonado, J. C., & Vincenzi, A. M. R. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107363 · Cited by 3 · see citing works
- Authors
- Durelli, V. H. S., Durelli, R. S., Borges, S. S., Endo, A. T., Eler, M. M., Dias, D. R. C., & Guimaraes, M. P.
- Affiliations
- Universidade Federal de Lavras (BR)
- Universidade Tecnológica Federal do Paraná (BR)
- Universidade de São Paulo (BR)
- Federal University of São João del-Rei (BR)
- ORCID
- Venue
- IEEE Transactions on Reliability
- Published
- 2019-09
- Volume
- 68
- Issue
- 3
- Pages
- 1189-1212
- ISSN
- 0018-9529 (ISSN-L)
- 1558-1721
- OpenAlex
- W2912899164
- Cited by
- 207
- References
- 93
- Citations by year
- 2026: 13 · 2025: 42 · 2024: 33 · 2023: 28 · 2022: 36 · 2021: 23 · 2020: 21 · 2019: 5 · 2018: 1 · 2017: 1 · 2015: 2 · 2013: 1 · 2012: 1
- FWCI
- 51.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Reliability (publishedVersion)
- Funders
- CNPq/Brazil (420363/2018-1)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- machine learning, software testing, test automation, test case generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
