When and what to automate in software testing? A multi-vocal literature review
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Within Software Engineering (SE), MLRs are being undertaken as exploratory studies and are seen as beneficial due to the practical nature of the field.
“Within Software Engineering (SE), MLRs are being undertaken as exploratory studies [46] [47] and are seen as beneficial due to the practical nature of the field [43].”
From DevSecOps practices and tools (2025) · cites this paper as [47]
Cites
Qualitative Data Analysis: A Methods Sourcebook
Miles, M. B., Huberman, A. M., & Saldaña, J. (1988) · Cited by 20,701 · see citing works
The Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · see citing works
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 2× · see citing works
“Snowballing, in this context, refers to using the reference list of a paper (backward snowballing) or the citations to the paper to identify additional papers (forward) [78].”
BackgroundMethodSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 3× · see citing works
“[68] proposed the classification of contributions into: method/technique, tool, model, metric and process.”
BackgroundMethod(2008) · Sage eBooks · doi:10.4135/9781412953948.n339 · Cited by 3,237 · see citing works
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669 · see citing works
Experimentation in Software Engineering: An Introduction
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2011) · Lund University Publications (Lund University) · Cited by 2,214 · see citing works
Publication bias in meta-analysis: Prevention, assessment and adjustments
Viechtbauer, W. (2007) · Psychometrika · doi:10.1007/s11336-006-1450-y · Cited by 1,507 · see citing works
Regression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing Verification and Reliability · doi:10.1002/stv.430 · Cited by 1,330 · see citing works
Principles and methods of testing finite state machines-a survey
Lee, D., & Yannakakis, M. (1996) · Proceedings of the IEEE · doi:10.1109/5.533956 · Cited by 1,230 · see citing works
Introduction to Software Testing
Bierig, R., Brown, S., Galván, E., & Timoney, J. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108974073.004 · Cited by 1,056 · see citing works
Recommended Steps for Thematic Synthesis in Software Engineering
Cruzes, D. S., & Dyba, T. (2011) · 2011 International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.36 · Cited by 960 · see citing works
Searching for grey literature for systematic reviews: challenges and benefits
Mahood, Q., Van Eerd, D., & Irvin, E. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1106 · Cited by 814 · see citing works
(2014) · Cited by 762 · see citing works
Experimentation in software engineering: An introduction
(2000) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(00)90203-7 · Cited by 696 · see citing works
Grey literature in meta-analyses of randomized trials of health care interventions
Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000010.pub3 · Cited by 598 · cited 1× · see citing works
“[34] conducted a review of five studies, in the area of evidence-based medicine, comparing the effect of the inclusion or exclusion of „grey‟ literature in meta-analyses of randomized medical trials.”
MethodWhyte, W. F., Greenwood, D. J., & Lazes, P. M. (1989) · American Behavioral Scientist · doi:10.1177/0002764289032005003 · Cited by 511 · see citing works
Ali, S., Briand, L., Hemmati, H., & Panesar-Walawege, R. K. (2009) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.52 · Cited by 452 · see citing works
An exploration of technical debt
Tom, E., Aurum, A., & Vidgen, R. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2012.12.052 · Cited by 438 · see citing works
State‐of‐the‐Evidence Reviews: Advantages and Challenges of Including Grey Literature
Benzies, K. M., Premji, S., Hayden, K. A., & Serrett, K. (2006) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1741-6787.2006.00051.x · Cited by 408 · see citing works
Research synthesis in software engineering: A tertiary study
Cruzes, D. S., & Dybå, T. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.01.004 · Cited by 339 · see citing works
A survey on model-based testing approaches
Neto, A. C. D., Subramanyan, R., Vieira, M., & Travassos, G. H. (2007) · doi:10.1145/1353673.1353681 · Cited by 306 · see citing works
A method for evaluating rigor and industrial relevance of technology evaluations
Ivarsson, M., & Gorschek, T. (2010) · Empirical Software Engineering · doi:10.1007/s10664-010-9146-4 · Cited by 280 · see citing works
Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise
Arisholm, E., Gallis, H., Dybå, T., & Sjøberg, D. I. K. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.17 · Cited by 274 · see citing works
Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method
Ogawa, R. T., & Malen, B. (1991) · Review of Educational Research · doi:10.3102/00346543061003265 · Cited by 247 · cited 2× · see citing works
“A 1991 paper [29] discussed rigor in MLRs and proposed a method based on exploratory case study to conduct MLRs rigorously.”
BackgroundMethodA systematic mapping study of software product lines testing
Neto, P. A. D. M. S., Machado, I., McGregor, J. D., de Almeida, E. S., & Meira, S. R. D. L. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.003 · Cited by 219 · see citing works
Garousi, V., Felderer, M., & Mäntylä, M. V. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916008 · Cited by 210 · see citing works
Self-citationLessons Learned in Software Testing
Kaner, C., Bach, J., & Pettichord, B. (2001) · Cited by 197 · see citing works
A systematic literature review to identify and classify software requirement errors
Walia, G., & Carver, J. C. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.01.004 · Cited by 194 · see citing works
The financial aspect of managing technical debt: A systematic literature review
Ampatzoglou, A., Ampatzoglou, A., Chatzigeorgiou, A., & Avgeriou, P. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.04.001 · Cited by 186 · see citing works
Rafi, D. M., Moses, K. R. K., Petersen, K., & Mäntylä, M. (2012) · International Workshop on Automation of Software Test (AST) · doi:10.1109/iwast.2012.6228988 · Cited by 170 · see citing works
Self-citationTest case prioritization: a systematic mapping study
Catal, C., & Mishra, D. (2012) · Software Quality Journal · doi:10.1007/s11219-012-9181-z · Cited by 168 · see citing works
Graphical user interface (GUI) testing: Systematic mapping and repository
Banerjee, I., Nguyen, B. N., Garousi, V., & Memon, A. M. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.03.004 · Cited by 168 · see citing works
Self-citationA survey of software testing practices in Canada
Garousi, V., & Zhi, J. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2012.12.051 · Cited by 167 · see citing works
Self-citationTesting scientific software: A systematic literature review
Kanewala, U., & Bieman, J. M. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.05.006 · Cited by 150 · see citing works
Rafi, D. M., Moses, K. R. K., Petersen, K., & Mäntylä, M. (2012) · Automation of Software Test · doi:10.5555/2663608.2663616 · Cited by 139 · see citing works
Self-citationGrey Literature and Systematic Reviews
Hopewell, S., Clarke, M. J., & Mallett, S. (2005) · doi:10.1002/0470870168.ch4 · Cited by 138 · see citing works
Software Testing: Principles and Practices
Desikan, S., & Ramesh, G. (2005) · Cited by 137 · see citing works
2009 IEEE International Advance Computing Conference
(2009) · doi:10.1109/iacc14940.2009 · Cited by 129 · see citing works
Automated Defect Prevention: Best Practices in Software Management
Huizinga, D. M., & Kolawa, A. (2007) · Cited by 121 · see citing works
Testing and verification in service‐oriented architecture: a survey
Bozkurt, M., Harman, M., & Hassoun, Y. (2012) · Software Testing Verification and Reliability · doi:10.1002/stvr.1470 · Cited by 120 · see citing works
Web application testing: A systematic literature review
Doğan, S., Betin-Can, A., & Garousi, V. (2014) · Journal of Systems and Software · doi:10.1016/j.jss.2014.01.010 · Cited by 113 · see citing works
Self-citationReAssert: Suggesting Repairs for Broken Unit Tests
Daniel, B., Jagannath, V., Dig, D., & Marinov, D. (2009) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2009.17 · Cited by 109 · see citing works
A systematic mapping study of web application testing
Garousi, V., Mesbah, A., Betin-Can, A., & Mirshokraie, S. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.02.006 · Cited by 108 · see citing works
Self-citationWhittaker, J. A. (2010) · doi:10.1145/1852666.1852675 · Cited by 90 · see citing works
A systematic mapping study on the combination of static and dynamic quality assurance techniques
Elberzhager, F., Münch, J., & Nha, V. T. N. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.06.003 · Cited by 86 · see citing works
A survey of software engineering practices in Turkey
Garousi, V., Coşkunçay, A., Betin-Can, A., & Demirörs, O. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.06.036 · Cited by 85 · see citing works
Self-citationAutomated Acceptance Testing: A Literature Review and an Industrial Case Study
Haugset, B., & Hanssen, G. K. (2008) · Agile Conference · doi:10.1109/agile.2008.82 · Cited by 85 · see citing works
Empirical Observations on Software Testing Automation
Karhu, K., Repo, T., Taipale, O., & Smolander, K. (2009) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2009.16 · Cited by 80 · see citing works
Software test-code engineering: A systematic mapping
Garousi, V., Amannejad, Y., & Betin-Can, A. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.06.009 · Cited by 76 · see citing works
Self-citationA View of the Dynamic Software Product Line Landscape
Bencomo, N., Hallsteinsen, S. O., & de Almeida, E. S. (2012) · Computer · doi:10.1109/mc.2012.292 · Cited by 75 · see citing works
UML behavioral model based test case generation
Shirole, M. R., & Kumar, R. (2013) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/2492248.2492274 · Cited by 71 · see citing works
Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality
Dustin, E., Garrett, T., & Gauf, B. (2009) · Cited by 71 · see citing works
Synthesizing evidence in software engineering research
Cruzes, D. S., & Dybå, T. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852788 · Cited by 70 · see citing works
Just Enough Software Test Automation
Mosley, D. J., & Posey, B. (2002) · International Linear Collider · Cited by 66 · see citing works
Introduction to software testing
(2008) · Choice Reviews Online · doi:10.5860/choice.45-6834 · Cited by 63 · see citing works
A Comprehensive Survey of Trends in Oracles for Software Testing
Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2013) · Cited by 59 · see citing works
How We Test Software at Microsoft
Page, A., & Johnston, K. (2008) · International Linear Collider · Cited by 54 · see citing works
The educational value of mapping studies of software engineering literature
Kitchenham, B. A., Pearl Brereton, A., & Budgen, D. (2010) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1806799.1806887 · Cited by 51 · see citing works
Trade-off between automated and manual software testing
Taipale, O., Kasurinen, J., Karhu, K., & Smolander, K. (2011) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-011-0065-6 · Cited by 48 · see citing works
A Systematic Literature Review of Software Process Improvement in Small and Medium Web Companies
Sulayman, M., & Mendes, E. (2009) · Communications in computer and information science · doi:10.1007/978-3-642-10619-4_1 · Cited by 43 · see citing works
A systematic mapping study on non-functional search-based software testing
Afzal, W., Torkar, R., & FELDT, R. H. (2008) · Cited by 38 · see citing works
Amannejad, Y., Garousi, V., Irving, R., & Sahaf, Z. (2014) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2014.34 · Cited by 36 · see citing works
Self-citationA systematic review on fault based mutation testing techniques and tools for Aspect-J programs
Singh, P. K., Sangwan, O. P., & Sharma, A. (2013) · IEEE International Advance Computing Conference · doi:10.1109/iadcc.2013.6514441 · Cited by 30 · see citing works
Sahaf, Z., Garousi, V., Pfahl, D., Irving, R., & Amannejad, Y. (2014) · International Conference on Software and System Process · doi:10.1145/2600821.2600832 · Cited by 29 · see citing works
Self-citationGarousi, V., & Herkiloğlu, K. (2016) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2016.7 · Cited by 19 · see citing works
Self-citationClassification and trend analysis of UML books (1997–2009)
Garousi, V. (2011) · Software & Systems Modeling · doi:10.1007/s10270-011-0189-9 · Cited by 13 · see citing works
Self-citationKuļešovs, I. (2015) · Environment Technology Resources Proceedings of the International Scientific and Practical Conference · doi:10.17770/etr2015vol3.187 · Cited by 12 · see citing works
Yasin, A., & Hasnain, M. (2012) · Cited by 11 · see citing works
Oakland, C. (2015) · Nursing Management · doi:10.7748/nm.22.2.15.s16 · Cited by 10 · see citing works
Test Cost-Effectiveness and Defect Density: A Case Study on the Android Platform
Garousi, V., Kotchorek, R., & Smith, M. R. (2013) · Advances in computers · doi:10.1016/b978-0-12-408094-2.00005-9 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
3 of the 74 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831 · see citing works
Self-citationAmpatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., & Chatzigeorgiou, A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.10.006 · Cited by 305 · see citing works
Garousi, V., Petersen, K., & Özkan, B. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.07.006 · Cited by 242 · see citing works
Self-citationDevSecOps: A Multivocal Literature Review
Myrbakken, H., & Colomo-Palacios, R. (2017) · Communications in Computer and Information Science · doi:10.1007/978-3-319-67383-7_2 · Cited by 205 · see citing works
Smells in software test code: A survey of knowledge in industry and academia
Garousi, V., & Küçük, B. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.12.013 · Cited by 189 · see citing works
Self-citationGiray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
Hijji, M., & Alam, G. (2021) · IEEE Access · doi:10.1109/access.2020.3048839 · Cited by 134 · see citing works
Software-testing education: A systematic literature mapping
Garousi, V., Rainer, A., Lauvås, P., & Arcuri, A. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110570 · Cited by 121 · see citing works
Self-citationAligning software engineering education with industrial needs: A meta-analysis
Garousi, V., Giray, G., Tüzün, E., Catal, C., & Felderer, M. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.06.044 · Cited by 119 · see citing works
Self-citationSoftware test maturity assessment and test process improvement: A multivocal literature review
Garousi, V., Felderer, M., & Hacaloğlu, T. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.01.001 · Cited by 105 · see citing works
Self-citationA Multi-Vocal Review of Security Orchestration
Islam, C., Babar, M. A., & Nepal, . (2019) · ACM Computing Surveys · doi:10.1145/3305268 · Cited by 94 · see citing works
Testing embedded software: A survey of the literature
Garousi, V., Felderer, M., Karapıçak, Ç. M., & Yılmaz, U. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.06.016 · Cited by 81 · see citing works
Self-citationNLP-assisted software testing: A systematic mapping of the literature
Garousi, V., Bauer, S., & Felderer, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106321 · Cited by 81 · see citing works
Self-citationPutta, A., Paasivaara, M., & Lassenius, C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03673-7_24 · Cited by 73 · see citing works
Simulation for Robotics Test Automation: Developer Perspectives
Afzal, A., Katz, D. S., Goues, C. L., & Timperley, C. S. (2021) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst49551.2021.00036 · Cited by 73 · see citing works
Relationship of DevOps to Agile, Lean and Continuous Deployment
Lwakatare, L. E., Kuvaja, P., & Oivo, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49094-6_27 · Cited by 71 · see citing works
Security in microservice-based systems: A Multivocal literature review
Pereira-Vale, A., Fernández, E. B., Monge, R., Hernán Astudillo, R., & Márquez, G. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102200 · Cited by 70 · see citing works
Automated functional testing of mobile applications: a systematic mapping study
Tramontana, P., Amalfitano, D., Amatucci, N., & Fasolino, A. R. (2018) · Software Quality Journal · doi:10.1007/s11219-018-9418-6 · Cited by 69 · see citing works
Test Automation: Not Just for Test Execution
Garousi, V., & Elberzhager, F. (2017) · IEEE Software · doi:10.1109/ms.2017.34 · Cited by 69 · see citing works
Self-citationA survey on software testability
Garousi, V., Felderer, M., & Kılıçaslan, F. N. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.12.003 · Cited by 64 · see citing works
Self-citationBenefitting from the Grey Literature in Software Engineering Research
Garousi, V., Felderer, M., Mäntylä, M., & Rainer, A. (2020) · Springer eBooks · doi:10.1007/978-3-030-32489-6_14 · Cited by 62 · see citing works
Self-citationbin Ali, N., & Usman, M. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.02.002 · Cited by 58 · see citing works
Otaduy, I., & Díaz, Ó. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.01.002 · Cited by 57 · see citing works
Sarker, O., Jayatilaka, A., Haggag, S., Liu, C., & Babar, M. A. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111899 · Cited by 62 · see citing works
Exploring the industry's challenges in software testing: An empirical study
Garousi, V., Felderer, M., Kuhrmann, M., Herkiloğlu, K., & Eldh, S. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2251 · Cited by 54 · see citing works
Self-citationA Taxonomy of Challenges for Self-Sovereign Identity Systems
Satybaldy, A., Ferdous, M. S., & Nowostawski, M. (2024) · IEEE Access · doi:10.1109/access.2024.3357940 · Cited by 51 · see citing works
Practical relevance of software engineering research: synthesizing the community’s voice
Garousi, V., Borg, M., & Oivo, M. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09803-0 · Cited by 48 · see citing works
Self-citationPrates, L., & Pereira, R. (2025) · International Journal of Information Security · doi:10.1007/s10207-024-00914-z · Cited by 49 · see citing works
“Within Software Engineering (SE), MLRs are being undertaken as exploratory studies [46] [47] and are seen as beneficial due to the practical nature of the field [43].”
Choosing the Right Test Automation Tool
Raulamo-Jurvanen, P., Mäntylä, M., & Garousi, V. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084252 · Cited by 46 · see citing works
Self-citationVarela‐Vaca, Á. J., & Quintero, A. M. R. (2021) · ACM Computing Surveys · doi:10.1145/3423166 · Cited by 46 · see citing works
AI-based Test Automation: A Grey Literature Analysis
Ricca, F., Marchetto, A., & Stocco, A. (2021) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw52544.2021.00051 · Cited by 45 · see citing works
A Review on Tools, Mechanics, Benefits, and Challenges of Gamified Software Testing
Fulcini, T., Coppola, R., Ardito, L., & Torchiano, M. (2023) · ACM Computing Surveys · doi:10.1145/3582273 · Cited by 45 · see citing works
Garousi, V., Özkan, R., & Betin-Can, A. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.06.007 · Cited by 40 · see citing works
Self-citationOperationalizing Human Values in Software Engineering: A Survey
Shahin, M., Hussain, W., Nurwidyantoro, A., Perera, H., Shams, R. A., Grundy, J., & Whittle, J. (2022) · IEEE Access · doi:10.1109/access.2022.3190975 · Cited by 40 · see citing works
Video Game Development in a Rush: A Survey of the Global Game Jam Participants
Borg, M., Garousi, V., Mahmoud, A., Olsson, T., & Stålberg, O. (2020) · IEEE Transactions on Games · doi:10.1109/tg.2019.2910248 · Cited by 38 · see citing works
Self-citationModel-driven architecture based testing: A systematic literature review
Uzun, B., & Teki̇nerdoğan, B. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.05.004 · Cited by 37 · see citing works
Bendler, D., & Felderer, M. (2022) · ACM Transactions on Computing Education · doi:10.1145/3573205 · Cited by 35 · see citing works
A Decade of Intelligent Software Testing Research: A Bibliometric Analysis
Boukhlif, M., Hanine, M., & Kharmoum, N. (2023) · Electronics · doi:10.3390/electronics12092109 · Cited by 37 · see citing works
Understanding Hypotheses Engineering in Software Startups through a Gray Literature Review
Melegati, J., Guerra, E., & Wang, X. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106465 · Cited by 35 · see citing works
Wang, Y., Mäntylä, M., Liu, Z., & Markkula, J. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111259 · Cited by 34 · see citing works
Self-citationSoftware-as-a-Service Security Challenges and Best Practices: A Multivocal Literature Review
Humayun, M., Niazi, M., Almufareh, M. F., Jhanjhi, N. Z., Mahmood, S., & Alshayeb, M. (2022) · Applied Sciences · doi:10.3390/app12083953 · Cited by 34 · see citing works
Rainer, A., & Williams, A. H. (2019) · Journal of Software Evolution and Process · doi:10.1002/smr.2197 · Cited by 32 · see citing works
How Should Software Engineering Secondary Studies Include Grey Material?
Kitchenham, B. A., Madeyski, L., & Budgen, D. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3165938 · Cited by 30 · see citing works
Towards automated video game testing
Politowski, C., Guéhéneuc, Y., & Petrillo, F. D. S. (2022) · doi:10.1145/3524494.3527627 · Cited by 30 · see citing works
Ali, S., Baseer, S., Abbasi, I. A., Alouffi, B., Alosaimi, W. A., & Huang, J. (2022) · Soft Computing · doi:10.1007/s00500-022-07062-3 · Cited by 29 · see citing works
Comparing automated visual GUI testing tools: an industrial case study
Garousi, V., Afzal, W., Çağlar, A., Işık, İ. B., Baydan, B., Çaylak, S., Boyraz, A. Z., Yolaçan, B., & Herkiloğlu, K. (2017) · doi:10.1145/3121245.3121250 · Cited by 28 · see citing works
Self-citationWeb Test Automation: Insights from the Grey Literature
Ricca, F., & Stocco, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67731-2_35 · Cited by 28 · see citing works
Processes, challenges and recommendations of Gray Literature Review: An experience report
Zhang, H., Mao, R., Huang, H., Dai, Q., Zhou, X., Shen, H., & Rong, G. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106607 · Cited by 26 · see citing works
Assessing test artifact quality—A tertiary study
Tran, H. K. V., Unterkalmsteiner, M., Börstler, J., & bin Ali, N. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106620 · Cited by 26 · see citing works
Test flakiness’ causes, detection, impact and responses: A multivocal review
Tahir, A., Rasheed, S., Dietrich, J. B., Hashemi, N., & Zhang, L. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111837 · Cited by 24 · see citing works
Serious games for teaching agile methods: A review of multivocal literature
Rodríguez, G., Caino, P. C. G., & Resett, S. A. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22430 · Cited by 24 · see citing works
Generation-based Code Review Automation: How Far Are We?
Zhou, X., Kim, K., Xu, B., Han, D., He, J., & Lo, D. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00036 · Cited by 29 · see citing works
Microservice-based projects in agile world: A structured interview
Ünlü, H., Kennouche, D. E., Soylu, G. K., & Demirörs, O. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107334 · Cited by 23 · see citing works
A Study on the Challenges of Using Robotics Simulators for TestingPreprint
Afzal, A., Katz, D. S., Goues, C. L., & Timperley, C. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.07368 · Cited by 31 · see citing works
Estimating Return on Investment for GUI Test Automation Frameworks
Dobslaw, F., FELDT, R. H., Michaelsson, D., Haar, P., Neto, F. G. D. O., & Torkar, R. (2019) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2019.00035 · Cited by 22 · see citing works
Ali, S., Ullah, N., Abrar, M. F., Yang, Z., & Huang, J. (2020) · Scientific Programming · doi:10.1155/2020/6597316 · Cited by 20 · see citing works
Kıraç, F., Aktemur, B., & Sözer, H. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.06.023 · Cited by 19 · see citing works
Garousi, V., & Yildirim, E. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00042 · Cited by 19 · see citing works
Self-citationMadeyski, L., & Stradowski, S. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112360 · Cited by 18 · see citing works
Scaling agile software development to large and globally distributed large-scale organizations
Putta, A. (2018) · IEEE International Conference on Global Software Engineering (ICGSE) · doi:10.1145/3196369.3196386 · Cited by 18 · see citing works
Practitioner Evaluations on Software Testing Tools
Raulamo-Jurvanen, P., Hosio, S. J., & Mäntylä, M. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319018 · Cited by 18 · see citing works
Self-citationYadav, R. S., Kaya, S. K., Pant, A., & Tiwari, A. (2023) · Global Knowledge, Memory and Communication · doi:10.1108/gkmc-04-2023-0128 · Cited by 17 · see citing works
Rainer, A., & Williams, A. H. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.10.007 · Cited by 17 · see citing works
On the Evolution of Keyword-Driven Test Suites
Rwemalika, R., Kintis, M., Papadakis, M., Traon, Y. L., & Lorrach, P. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00040 · Cited by 17 · see citing works
Naveed, A., Zhuparova, A., Ahmad, N., & Aghdam, R. F. (2023) · Total Quality Management & Business Excellence · doi:10.1080/14783363.2022.2157713 · Cited by 16 · see citing works
Test activities in the continuous integration and delivery pipeline
Mårtensson, T., Ståhl, D., & Bosch, J. (2019) · Journal of Software Evolution and Process · doi:10.1002/smr.2153 · Cited by 16 · see citing works
Li, T., Zhang, X., Wang, Y., Zhou, Q., Wang, Y., & Dong, F. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107477 · Cited by 15 · see citing works
Beyond Technical Debt Unravelling Organisational Debt Concept
Ahmad, M. O., & Al-Baik, O. (2024) · ACM Symposium on Applied Computing · doi:10.1145/3605098.3635913 · Cited by 15 · see citing works
A multi-year grey literature review on AI-assisted test automation
Ricca, F., Marchetto, A., & Stocco, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107799 · Cited by 15 · see citing works
A Systematic Review of Software Testing Using Evolutionary Techniques
Mishra, D. B., Mishra, R., Das, K. N., & Acharya, A. A. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-981-10-3322-3_16 · Cited by 15 · see citing works
Technical Debt Prioritization: Taxonomy, Methods Results, and Practical Characteristics
Pina, D. D. J., Goldman, A., & Tonin, G. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00034 · Cited by 15 · see citing works
Qualitative Surveys in Software Engineering Research: Definition, Critical Review, and Guidelines
Melegati, J., Conboy, K., & Graziotin, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3474173 · Cited by 14 · see citing works
Chaos Engineering: A Multi-Vocal Literature Review
Owotogbe, J., Kumara, I., van den Heuvel, W., & Tamburri, D. A. (2025) · ACM Computing Surveys · doi:10.1145/3777375 · Cited by 14 · see citing works
Lambiase, S., Catolino, G., Palomba, F., & Ferrucci, F. (2024) · ACM Computing Surveys · doi:10.1145/3704806 · Cited by 14 · see citing works
Defect Categorization in Compilers: A Multi-vocal Literature Review
Rahman, A., Bose, D. B., Barsha, F. L., & Pandita, R. (2023) · ACM Computing Surveys · doi:10.1145/3626313 · Cited by 13 · see citing works
Test Automation Process Improvement in a DevOps Team: Experience Report
Wang, Y., Pyhajarvi, M., & Mantyla, M. V. (2020) · 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw50294.2020.00057 · Cited by 12 · see citing works
Self-citationImproving test automation maturity: A multivocal literature review
Wang, Y., Mäntylä, M., Liu, Z., Markkula, J., & Raulamo-Jurvanen, P. (2022) · Software Testing Verification and Reliability · doi:10.1002/stvr.1804 · Cited by 12 · see citing works
Self-citationWilliams, A. H., & Rainer, A. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319013 · Cited by 11 · see citing works
Testing in a DevOps Era: Perceptions of Testers in Norwegian Organisations
Cruzes, D. S., Kristin, M., & Marczak, S. D. S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-24305-0_33 · Cited by 11 · see citing works
The Use of Grey Literature Review as Evidence for Practitioners
Kamei, F. (2019) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3356773.3356797 · Cited by 11 · see citing works
Quality Assessment Methods for Textual Conversational Interfaces: A Multivocal Literature Review
Coppola, R., & Ardito, L. (2021) · Information · doi:10.3390/info12110437 · Cited by 16 · see citing works
Al-Baik, O., Abu-Alhaija, M., Abdeljaber, H. A. M., & Ahmad, M. O. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308183 · Cited by 10 · see citing works
Automated Testing of Simulation Software in the Aviation Industry: An Experience Report
Garousi, V., Tasli, S., Sertel, O., Tokgoz, M., Herkiloğlu, K., Arkin, H. F. E., & Bilir, O. (2018) · IEEE Software · doi:10.1109/ms.2018.227110307 · Cited by 10 · see citing works
Self-citationDo software engineering practitioners cite research on software testing in their online articles?
Williams, A. H. (2018) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3210459.3210475 · Cited by 10 · see citing works
Garousi, V., & Rainer, A. (2020) · IEEE Software · doi:10.1109/ms.2020.3022931 · Cited by 10 · see citing works
Self-citationAutomated Software Testing on Mobile Applications: A Review with Special Focus on Android Platform
Musthafa, F. N., Mansur, S. M. H., & Wibawanto, A. (2020) · International Conference on Advances in ICT for Emerging Regions · doi:10.1109/icter51097.2020.9325445 · Cited by 10 · see citing works
How to Create a Software Ecosystem? A Partnership Meta-Model and Strategic Patterns
Belo, Í., & Alves, C. F. (2021) · Information · doi:10.3390/info12060240 · Cited by 10 · see citing works
Snapshot testing in practice: Benefits and drawbacks
Cruz, V. P. G., Rocha, H., & Valente, M. T. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111797 · Cited by 9 · see citing works
Improving Software Automation Testing Using Jenkins, and Machine Learning Under Big Data
Stouky, A., Jaoujane, B., Daoudi, R., & Chaoui, H. (2018) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-319-98752-1_10 · Cited by 9 · see citing works
Reanzi, S. R. D., & Thangaiah, P. R. J. (2021) · International Journal of Engineering Business Management · doi:10.1177/18479790211062044 · Cited by 9 · see citing works
Holacracy In Software Development Teams: A Multivocal Literature Review
Bhandari, R., & Colomo‐Palacios, R. (2019) · International Conference on Computational Science and Its Applications · doi:10.1109/iccsa.2019.00013 · Cited by 9 · see citing works
A Self-assessment Instrument for Assessing Test Automation Maturity
Wang, Y., Mäntylä, M., Eldh, S., Markkula, J., Wiklund, K., Kairi, T., Raulamo-Jurvanen, P., & Haukinen, A. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319020 · Cited by 8 · see citing works
Self-citationA Survey of the Software Test Methods and Identification of Critical Success Factors for Automation
Bhargavi, S. M. B., & Suma, V. (2022) · SN Computer Science · doi:10.1007/s42979-022-01297-5 · Cited by 8 · see citing works
Butt, S., Khan, S. U. R., Hussain, S., & Wang, W. (2022) · Data & Knowledge Engineering · doi:10.1016/j.datak.2022.102111 · Cited by 7 · see citing works
de Andrade, S. A., Nunes, F. L. S., & Delamaro, M. E. (2023) · Software Testing Verification and Reliability · doi:10.1002/stvr.1863 · Cited by 7 · see citing works
Decision Support for Selecting Tools for Software Test Automation
Raulamo-Jurvanen, P. (2017) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3011286.3011304 · Cited by 7 · see citing works
Product managers in software startups: A grounded theory
Melegati, J., Wiese, I., Guerra, E., Chanin, R. M., Aldaeej, A. A., Mikkonen, T., Prikladnicki, R., & Wang, X. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107516 · Cited by 6 · see citing works
Do software engineering practitioners cite software testing research in their online articles?
Williams, A. H., & Rainer, A. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319708 · Cited by 6 · see citing works
Felderer, M., & Garousi, V. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-35510-4_1 · Cited by 6 · see citing works
Self-citationAbrar, M. F., Majeed, M. F., Saqib, M., Alferaidi, A., & Uddin, R. (2025) · IEEE Access · doi:10.1109/access.2025.3574200 · Cited by 5 · see citing works
- Authors
- Garousi, V., & Mäntylä, M. V.
- Affiliations
- Hacettepe University (TR)
- University of Oulu (FI)
- Corresponding author
- Garousi, Vahid
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2016-08
- Volume
- 76
- Pages
- 92-117
- ISSN
- 0950-5849 (ISSN-L)
- OpenAlex
- W2345039647
- Cited by
- 182
- References
- 74
- Citations by year
- 2026: 8 · 2025: 19 · 2024: 12 · 2023: 21 · 2022: 18 · 2021: 21 · 2020: 22 · 2019: 26 · 2018: 18 · 2017: 14 · 2016: 3
- FWCI
- 69.60
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Information and Software Technology (publishedVersion)
- Funders
- Hacettepe University
- Scientific and Technological Research Council of Turkey (115E805)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
- Keywords
- test automation, regression testing, economic factors, software maturity, test tools, human and organizational factors, grey literature
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
