Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 401–500 of 3,934
Continual Object Detection: A review of definitions, strategies, and challenges
Menezes, A. G., de Moura, G., Alves, C., & de Carvalho, A. C. P. L. F. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.01.041 · Cited by 104
Behutiye, W., Karhapää, P., López, L., Illa, X. B., Martínez‐Fernández, S., Vollmer, A. M., Rodríguez, P., Franch, X., & Oivo, M. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106225 · Cited by 101
“Backward snowballing uses the reference list of a paper to identify more studies, whereas forward snowballing uses the studies citing a given paper to identify additional studies [53].”
BackgroundMethodInfluentialCircular business models in the medical device industry: paths towards sustainable healthcare
Guzzo, D., de Carvalho, M. M., Balkenende, R., & Mascarenhas, J. (2020) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2020.104904 · Cited by 97
Detecting Medical Misinformation on Social Media Using Multimodal Deep Learning
Wang, Z., Yin, Z., & Argyris, Y. A. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3037027 · Cited by 86
“For the data collection, we adopted a snowballing method widely used for systematic literature reviews [51].”
MethodA systematic literature review and taxonomy of modern code review
Davila, N., & Nunes, I. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110951 · Cited by 115
“We then conducted a backward snowballing [161] using this reference list to identify additional primary studies.”
MethodResearch on Third-Party Libraries in Android Apps: A Taxonomy and Systematic Literature Review
Zhan, X., Liu, T., Fan, L., Li, L., Chen, S., Luo, X., & Liu, Y. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3114381 · Cited by 77
“In accordance with international academic rules [44], [45], publications that are less than five pages in IEEE/ACM-like double-column format or less than seven pages in LNCS-”
BackgroundMethodA Literature Review of Using Machine Learning in Software Development Life Cycle Stages
Shafiq, S., Mashkoor, A., Mayr‐Dorn, C., & Egyed, A. F. (2021) · IEEE Access · doi:10.1109/access.2021.3119746 · Cited by 110
Virtual luxury in the metaverse: NFT-enabled value recreation in luxury brands
Bao, W., Hudders, L., Yu, S., & Beuckels, E. (2024) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2024.01.002 · Cited by 96
Critical scientific and environmental literacies: a systematic and critical review
Guerrero, G., & Sjöström, J. (2024) · Studies in Science Education · doi:10.1080/03057267.2024.2344988 · Cited by 163
“Finally, we used snowballing approach as a complementary process to identify relevant literature (Wohlin, 2014).”
Information Sharing in Cybersecurity: A Review
Pala, A., & Zhuang, J. (2019) · Decision Analysis · doi:10.1287/deca.2018.0387 · Cited by 91
Cyber Resilience and Incident Response in Smart Cities: A Systematic Literature Review
Ahmadi‐Assalemi, G., Al‐Khateeb, H., Epiphaniou, G., & Maple, C. R. (2020) · Smart Cities · doi:10.3390/smartcities3030046 · Cited by 104
“In addition to the set of studies produced through the search of the DLs, we applied a snowballing approach in our search strategy, as outlined by Wohlin [69], which produced a further set of relevant studies.”
MethodA Review of Testing Object-Based Environment Perception for Safe Automated Driving
Hoss, M., Scholtes, M., & Eckstein, L. (2022) · Automotive Innovation · doi:10.1007/s42154-021-00172-y · Cited by 81
“The methods of this literature review are loosely inspired by the guidelines for systematic literature reviews [23] and snowballing search [24], and respective example applications of both in Refs.”
MethodThe do’s and don’ts of infrastructure code: A systematic gray literature review
Kumara, I., Garriga, M., Romeu, A. U., Di Nucci, D., Palomba, F., Tamburri, D. A., & van den Heuvel, W.-J. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106593 · Cited by 70
“Items found using the search terms have been assessed thoroughly based on various dimensions of quality [56].”
Strandberg, K., Nowdehi, N., & Olovsson, T. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3188340 · Cited by 87
“We then performed a snowballing approach [34].”
Text-Mining Techniques and Tools for Systematic Literature Reviews: A Systematic Literature Review
Feng, L., Chiam, Y. K., & Lo, S. K. (2017) · 2017 24th Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2017.10 · Cited by 70
“We used Google Scholar as a retrieval facility to conduct snowballing in our search process to identify new papers [19].”
MethodVoice Assistants and Artificial Intelligence in Education
Terzopoulos, G., & Satratzemi, M. (2019) · Proceedings of the 9th Balkan Conference on Informatics · doi:10.1145/3351556.3351588 · Cited by 70
“In order to retrieve sufficient and high-quality papers regarding uses of voice assistants and smart speakers, the snowball technique as described in [ 12] was used.”
MethodA systematic literature review of software requirements reuse approaches
Irshad, M., Petersen, K., & Poulding, S. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.09.009 · Cited by 76
MTD, Where Art Thou? A Systematic Review of Moving Target Defense Techniques for IoT
Navas, R. E., Cuppens, F., Cuppens-Boulahia, N., Toutain, L., & Papadopoulos, G. Z. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3040358 · Cited by 79
“3) Snowballing: Snowballing is a search technique that identifies potential additional articles to include in a systematic literature review [31].”
BackgroundSecurity in microservice-based systems: A Multivocal literature review
Pereira-Vale, A., Fernández, E. B., Monge, R., Hernán Astudillo, R., & Márquez, G. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102200 · Cited by 104
Grey Literature in Software Engineering: A critical review
Kamei, F., Wiese, I., Lima, C., Polato, I., Nepomuceno, V., Ferreira, W., Ribeiro, M., Pena, C., Cartaxo, B., Pinto, G., et al. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106609 · Cited by 98
“We found that Kitchenham and Charters’ guidelines [16] were most used (67 studies) among Secondary Studies, followed by Wohlin’s guidelines used to conduct snowballing [42].”
MethodNeto, S. R. D. S., de Oliveira, T. T., Teixeira, I. V., de Oliveira, S. B. A., Sampaio, V. D. S., Lynn, T., & Endo, P. T. (2022) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0010061 · Cited by 85
What determines FinTech success?—A taxonomy-based analysis of FinTech success factors
Werth, O., Cardona, D. R., Torno, A., Breitner, M. H., & Muntermann, J. (2023) · Electronic Markets · doi:10.1007/s12525-023-00626-7 · Cited by 104
“Based on the guiding principles of Webster and Watson (2002) and Wohlin (2014), we performed both a keyword and a forward–backward literature search (Hinde & Spackman, 2015), which we further complemented with an author and similarity search as shown below: The determination of the representative…”
BackgroundMethodMraihi, T., Driss, O. B., & El-Haouzi, H. B. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121947 · Cited by 69
Second-generation systematic literature studies using snowballing
Wohlin, C. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916006 · Cited by 69
“The guidelines in [13] were intended to make snowballing more common in software engineering too.”
BackgroundMethodInfluentialSelf-citationBozhinoski, D., Di Ruscio, D., Malavolta, I., Pelliccione, P., & Crnkovic, I. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.02.021 · Cited by 65
“More technically, we performed a closed recursive backward and forward snowballing activity [30].”
MethodInfluentialAdoption of learning analytics in higher education institutions: A systematic literature review
Márquez, L., Henríquez, V., Chevreux, H., Scheihing, E., & Guerra, J. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13385 · Cited by 69
Yu, A. T. W., Yevu, S. K., & Nani, G. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.119493 · Cited by 81
A Survey of Anomaly Detection for Connected Vehicle Cybersecurity and Safety
Rajbahadur, G. K., Malton, A. J., Walenstein, A., & Hassan, A. E. (2018) · 2018 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2018.8500383 · Cited by 68
“To ensure reproducibility, our survey follows Wholin’s snowballing method [14] as follows.”
BackgroundMethodInfluentialDesign Thinking: Challenges for Software Requirements Elicitation
Martins, H. F., de Oliveira, A. C., Canedo, E. D., Kosloski, R. A. D., Paldês, R. Á., & Oliveira, E. C. (2019) · Information · doi:10.3390/info10120371 · Cited by 90
“During the SLR, the Planning, Conduct and Publication of Results were followed, as defined in the work presented by Kitchenham [35], together with the Manual Search and the Snowballing [36] in order to answer RQ.1.”
MethodInitiatives and challenges of using gamification in software engineering: A Systematic Mapping
Porto, D. D. P., de Jesus, G. M., Ferrari, F. C., & Fabbri, S. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110870 · Cited by 99
Christofi, M., Iaia, L., Marchesani, F., & Masciarelli, F. (2021) · International Marketing Review · doi:10.1108/imr-01-2021-0027 · Cited by 76
A Survey of Challenges in Spectrum-Based Software Fault Localization
Sarhan, Q. I., & Beszédes, Á. (2022) · IEEE Access · doi:10.1109/access.2022.3144079 · Cited by 96
“In this paper, the snowballing technique [36] was also used to reduce the risk of missing some relevant papers.”
MethodSoftware Engineering Knowledge Areas in Startup Companies: A Mapping Study
Klotins, E., Unterkalmsteiner, M., & Gorschek, T. (2015) · Lecture notes in business information processing · doi:10.1007/978-3-319-19593-3_22 · Cited by 80
Data Preparation for Software Vulnerability Prediction: A Systematic Literature Review
Croft, R., Xie, Y., & Babar, M. A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3171202 · Cited by 96
“lists of the set of included studies, respectively [33].”
BackgroundFramework for standardising carbon neutrality in building projects
Too, J., EJOHWOMU, O. A., Hui, F. K. P., Duffield, C. F., Bukoye, O. T., & Edwards, D. J. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.133858 · Cited by 71
Climate change and its impacts on glaciers and glacial lakes in Nepal Himalayas
Khadka, N., Chen, X., Sharma, S., & Shrestha, B. (2023) · Regional Environmental Change · doi:10.1007/s10113-023-02142-y · Cited by 95
“In addition, the snowballing technique (Wohlin 2014) was used for this review so that original contributions on the topic related to Nepal would not be disregarded.”
MethodTowards a Taxonomy of Microservices Architectures
Garriga, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-74781-1_15 · Cited by 96
“Then, we applied snowballing [12], by looking for relevant references included in the works previously found, in order to identify other potential works.”
MethodShe, Q., Wei, X., Nie, G., & Chen, D. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.07.021 · Cited by 69
Set of Usability Heuristics for Quality Assessment of Mobile Applications on Smartphones
da Costa, R. P., Canedo, E. D., de Sousa, R. T., Albuquerque, R. D. O., & Villalba, L. J. G. (2019) · IEEE Access · doi:10.1109/access.2019.2910778 · Cited by 85
Risks of modular integrated construction: A review and future research directions
Wuni, I. Y., Shen, Q., & Hwang, B. (2019) · Frontiers of Engineering Management · doi:10.1007/s42524-019-0059-7 · Cited by 67
Adoption of alternative fuel vehicle fleets – A theoretical framework of barriers and enablers
Mohammed, L., Niesten, E., & Gagliardi, D. (2020) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2020.102558 · Cited by 86
A taxonomy of service identification approaches for legacy software systems modernization
Abdellatif, M., Shatnawi, A., Mili, H., Moha, N., Boussaidi, G. E., Hecht, G., Privat, J., & Guéhéneuc, Y. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110868 · Cited by 102
Blockchain in humanitarian operations management: A review of research and practice
Hunt, K., Narayanan, A. N. L., & Zhuang, J. (2021) · Socio-Economic Planning Sciences · doi:10.1016/j.seps.2021.101175 · Cited by 80
Passing the Data Baton : A Retrospective Analysis on Data Science Work and Workers
Crisan, A., Fiore-Gartland, B., & Tory, M. K. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030340 · Cited by 93
“We used a “snowball approach” [85] to conduct our literature review.”
MethodSmartphone-Based Indoor Localization Systems: A Systematic Literature Review
Naser, R. S., Lam, M. C., Qamar, F., & Zaidan, B. B. (2023) · Electronics · doi:10.3390/electronics12081814 · Cited by 74
“In addition, manual search and snowballing [58] were deployed to integrate more studies.”
Badampudi, D., Wohlin, C., & Petersen, K. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.07.027 · Cited by 80
Self-citationBérubé, M., Poitras, S., Bastien, M., Laliberté, L.-A., Lacharité, A., & Gross, D. P. (2017) · Physiotherapy · doi:10.1016/j.physio.2017.05.002 · Cited by 79
“the reference list of a publication and the citations of the reference to identify additional papers (21).”
BackgroundThe Potential of Open Space for Enhancing Urban Seismic Resilience: A literature Review
Koren, D., & Rus, K. R. (2019) · Sustainability · doi:10.3390/su11215942 · Cited by 80
“In addition, a few documents were identified using the snowball approach [29].”
BackgroundHansen, U. E., Nygaard, I., Morris, M., & Robbins, G. (2020) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2020.109843 · Cited by 83
Big Data analytics in Agile software development: A systematic mapping study
Biesialska, K., Franch, X., & Muntés-Mulero, V. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106448 · Cited by 88
Surface Remeshing: A Systematic Literature Review of Methods and Research Directions
Khan, D., Plopski, A., Fujimoto, Y., Kanbara, M., Jabeen, G., Zhang, Y. J., Zhang, X., & Kato, H. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3016645 · Cited by 73
“In addition to SLR and manual search, we also used the snowballing concept [22] to include further studies.”
MethodMaritime Traffic as a Complex Network: a Systematic Review
Álvarez, N. G., Adenso-Díaz, B., & Calzada-Infante, L. (2021) · Networks and Spatial Economics · doi:10.1007/s11067-021-09528-7 · Cited by 75
“Once the first selection of papers has been made, the “snowballing” technique is applied (Wohlin 2014).”
MethodSalihu, C., Hussein, M., Mohandes, S. R., & Zayed, T. M. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.131460 · Cited by 76
Skills for the twin transition in manufacturing: A systematic literature review
Trevisan, A. H., Acerbi, F., Dukovska‐Popovska, I., Terzi, S., & Sassanelli, C. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.143603 · Cited by 109
A survey on software testability
Garousi, V., Felderer, M., & Kılıçaslan, F. N. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.12.003 · Cited by 89
Silva, R. J., & Tamburic, S. (2022) · Cosmetics · doi:10.3390/cosmetics9050090 · Cited by 63
“Upon selection of a relevant number of academic papers and articles, backward and forward snowballing techniques [16] were used to identify additional material.”
MethodFramework for Considering Productive Aging and Work
Schulte, P. A., Grosch, J. W., Scholl, J. C., & Tamers, S. L. (2018) · Journal of Occupational and Environmental Medicine · doi:10.1097/jom.0000000000001295 · Cited by 72
“A snowball literature review approach was used.(34) The authors first conducted a forward snowball approach, which began with the work of Butler and Gleason(30) and Rowe and Kahn(31) and Ilmarinen.”
MethodZiogas, I. A., Evangeliou, A. P., Giannis, D., Hayat, M. H., Mylonas, K. S., Tohme, S. T., Geller, D. A., Elias, N., Goyal, L., & Tsoulfas, G. (2021) · The Oncologist · doi:10.1002/onco.13638 · Cited by 68
“The reference lists of studies included in the systematic review were also hand-searched for missed studies using the systematic “snowballing” procedure guidelines [21].”
MethodIoannou, A., Tussyadiah, I. P., Miller, G. A., Li, S., & Weick, M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256822 · Cited by 85
“It has been suggested that researchers can achieve the best possible coverage of existing literature by using snowballing as a complementary approach to database search [19].”
BackgroundRequirements Engineering for Machine Learning: A Systematic Mapping Study
Villamizar, H., Escovedo, T., & Kalinowski, M. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00013 · Cited by 101
“In the next step, during the month of February, we applied backward and forward snowballing iteratively following the snowballing guidelines in [60].”
MethodInfluentialTransfer Learning With Time Series Data: A Systematic Mapping Study
Weber, M., Auch, M., Doblander, C., Mandl, P., & Jacobsen, H. (2021) · IEEE Access · doi:10.1109/access.2021.3134628 · Cited by 86
Strategies for Fostering a Genuine Feeling of Connection in Technologically Mediated Systems
Stepanova, E. R., Desnoyers-Stewart, J., Höök, K., & Riecke, B. E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517580 · Cited by 81
“We compiled the corpus of papers describing the design artifacts through a snowballing approach [188] by searching ACM Database and Google Scholar using keywords related to the experience of connection, such as connection, belonging,”
MethodIs Fragmentation a Threat to the Success of the Internet of Things?
Aly, M., Khomh, F., Guéhéneuc, Y., Washizaki, H., & Yacout, S. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2863180 · Cited by 80
“3) Round 3: Following existing guidelines [50], [52], we perform a snowball search using the reference list of our studies and citations obtained from the previous Round 2 to identify new studies and decide whether to include additional paper(s) or article(s); such technique helps us not to miss important and relevant…”
MethodAn Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability
Barón, M. M., Wyrich, M., & Wagner, S. (2020) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3382494.3410636 · Cited by 121
“Here, a single iteration of backward snowballing was applied to the papers remaining after going through all 2http://scholar.google.com/ 3https://dl.acm.org/ 4https://ieeexplore.ieee.org/ 5https://www.sciencedirect.com/ filtering steps as it is described by Wohlin [41].”
BackgroundMethodInfluentialA systematic literature review of cross-domain model consistency checking by model management tools
Torres, W., van den Brand, M., & Serebrenik, A. (2020) · Software & Systems Modeling · doi:10.1007/s10270-020-00834-1 · Cited by 78
“coverage of electronic sources of different research areas, and it has been used in multiple software engineering studies [52,70,82,93,100,150].”
BackgroundMethodA Survey on Botnets: Incentives, Evolution, Detection and Current Trends
Vu, S. N. T., Stege, M., El-Habr, P. I., Bang, J., & Dragoni, N. (2021) · Future Internet · doi:10.3390/fi13080198 · Cited by 87
“The methodology contains elements from both [37,38], which provide guidelines on how to write a systematic literature review and how to use snowball sampling for paper inclusion respectively.”
MethodA trait‐based framework for assessing the vulnerability of marine species to human impacts
Butt, N., Halpern, B. S., O’Hara, C. C., Allcock, A. L., Polidoro, B., Sherman, C. S., Byrne, M., Birkeland, C., Dwyer, R. G., Frazier, M. R., et al. (2022) · Ecosphere · doi:10.1002/ecs2.3919 · Cited by 72
“We used the snowball method (Wohlin, 2014) to review the literature, using search terms “marine,” “marine species,” “vulnerability assessment,” “traits,” “life-history traits” to further support and guide the development of the framework.”
MethodAuthentication and Authorization in Microservices Architecture: A Systematic Literature Review
de Almeida, M. G., & Canedo, E. D. (2022) · Applied Sciences · doi:10.3390/app12063023 · Cited by 87
“We also applied the “snowballing” process which aims to prevent relevant studies from being omitted [20].”
MethodCurriculum, Teaching and Learning, and Assessments for Introductory Programming Course
Mehmood, E., Abid, A., Farooq, M. S., & Nawaz, N. A. H. (2020) · IEEE Access · doi:10.1109/access.2020.3008321 · Cited by 79
“After performing quality assessment, we conducted back-ward snowballing [15] through reference list of each finalized study to extract papers.”
MethodEnergy resources and electricity generation in Arctic areas
de Witt, M., Stefánsson, H., Valfells, Á., & Larsen, J. N. (2021) · Renewable Energy · doi:10.1016/j.renene.2021.01.025 · Cited by 74
Laumbacher, H., Strasser, T., Knüttel, H., & Rosentritt, M. (2021) · Journal of Dentistry · doi:10.1016/j.jdent.2021.103723 · Cited by 75
“The electronic search was complemented by manual searches of the bibliographies of all included reviews for additional relevant articles and forward snowballing using PubMed functions “Cited by” and “Similar articles” [7].”
MethodUnderstanding embodied immersion in technology‐enabled embodied learning environments
Xu, X., Kang, J., & Yan, L. (2021) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12594 · Cited by 81
Hurmelinna‐Laukkanen, P., & Yang, J. (2021) · Research Policy · doi:10.1016/j.respol.2021.104417 · Cited by 77
Wuni, I. Y. (2022) · Sustainable Development · doi:10.1002/sd.2449 · Cited by 77
Machine Learning for Detecting Data Exfiltration: A Review
Sabir, B., Ullah, F., Babar, M. A., & Gaire, R. (2021) · ACM Computing Surveys · doi:10.1145/3442181 · Cited by 98
“It was then complemented with forward and backwards snowballing [55].”
MethodCode Smells Detection and Visualization: A Systematic Literature Review
dos Reis, J. P., Abreu, F. B. E., Carneiro, G. D. F., & Anslow, C. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09566-x · Cited by 100
Automated third-party library detection for Android applications
Zhan, X., Fan, L., Liu, T., et al. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416582 · Cited by 60
“To investigate existing TPL detection techniques, we first follow a well-defined Systematic Literature Review (SLR) methodology [1, 60] to find related research in this area.”
MethodIDEAS for Transforming Higher Education: An Overview of Ongoing Trends and Challenges
Ortiz, L. G., Clougher, D., Anderson, T. D., & Maina, M. F. (2021) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v22i2.5206 · Cited by 110
Centanni, M., Ricci, G. F., Girolamo, A. M. D., Romano, G., & Gentile, F. (2022) · Environmental Pollution · doi:10.1016/j.envpol.2022.120553 · Cited by 61
Nascita, A., Aceto, G., Ciuonzo, D., Montieri, A., Persico, V., & Pescapè, A. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3504955 · Cited by 102
“To conduct our study, we have adopted a systematic approach to exploring the literature, following the methodology outlined by Wohlin [5].”
Ouriques, R., Wnuk, K., Gorschek, T., & Svensson, R. B. (2019) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194019500153 · Cited by 70
“strated in the conceptual classification framework (see Section 5). 3.3. Search process and study selection The search process followed two approaches. According to Kitchenhan et al., [51] and Wohlin [52], combining different approaches is a way of having the best possible literature coverage. We applied automated sea…”
MethodInfluentialZiogas, I. A., Giannis, D., Economopoulos, K. P., Hayat, M. H., Montenovo, M. I., Matsuoka, L. K., & Alexopoulos, S. P. (2021) · Transplantation · doi:10.1097/tp.0000000000003539 · Cited by 66
“We also hand-searched the reference lists and previously published narrative and systematic reviews for potentially eligible, missed studies according to the systematic “snowball” methodology.(18) Data Tabulation and Extraction”
MethodGenerative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Security Mechanisms Used in Microservices-Based Systems: A Systematic Mapping
Pereira-Vale, A., Márquez, G., Hernán Astudillo, R., & Fernández, E. B. (2019) · Latin American Computer Conference (CLEI) · doi:10.1109/clei47609.2019.235060 · Cited by 66
“3) Snowballing process : We applied the “snowballing” process [13] aiming not to miss relevant studies.”
MethodMembrane technology for a sustainable copper mining industry: The Chilean paradigm
Santoro, S., Estay, H., Avci, A. H., Pugliese, L., Ruby‐Figueroa, R., García, A., Aquino, M., Nasirov, S., Straface, S., & Curcio, E. (2021) · Cleaner Engineering and Technology · doi:10.1016/j.clet.2021.100091 · Cited by 68
Microservice security: a systematic literature review
Berardi, D., Giallorenzo, S., Mauro, J., Melis, A., Montesi, F., & Prandini, M. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.779 · Cited by 100
“As the last (5th) step for the systematic literature review, we performed a backward snowballing process (Wohlin, 2014) with the objective of identifying additional relevant references for our study from the works cited by the already selected publications.”
MethodBlockchain-Based Identity Management Systems in Health IoT: A Systematic Review
Alamri, B., Crowley, K., & Richardson, I. (2022) · IEEE Access · doi:10.1109/access.2022.3180367 · Cited by 75
“To ensure that a wide range of literature was analyzed, a backward and forward snowballing approach was used [39].”
MethodImam, M., Bäına, K., Tabii, Y., Ressami, E. M., Adlaoui, Y., Benzakour, I., & Abdelwahed, E. H. (2023) · Sensors · doi:10.3390/s23094294 · Cited by 84
“The snowballing approach [61] called for the inclusion of all literature and an extension of the publication time in the second phase.”
Vasileiou, M., Kyrgiakos, L. S., Kleisiari, C., Lappas, P. Z., Tsinopoulos, C. D., Kleftodimos, G., Ntemou, A., Katerıs, D., Moulogianni, C., & Vlontzos, G. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-025-00948-0 · Cited by 118
“Based on the full-text assessment, 54 articles were included in this research, along with 7 publications that were found using snow-balling (Wohlin 2014).”
A Systematic Literature Review of iStar extensions
Gonçalves, E. J. T., Castro, J., Araújo, J., & Heineck, T. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.11.023 · Cited by 68
Ali, N. B., & Usman, M. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.02.002 · Cited by 58
Systematic Survey on Smart Home Safety and Security Systems Using the Arduino Platform
Sarhan, Q. I. (2020) · IEEE Access · doi:10.1109/access.2020.3008610 · Cited by 68
“To reduce the risk of missing some relevant papers, the snowballing search technique [20] was applied to the remaining papers.”
MethodJafar, U., Ab Aziz, M. J., Shukur, Z., & Hussain, H. A. (2022) · Sensors · doi:10.3390/s22197585 · Cited by 59
“The criteria enabled us to generate a collection of findings that we could then put via [121] snowballing process.”
BackgroundSustainable software engineering: Reflections on advances in research and practice
Venters, C. C., Capilla, R., Nakagawa, E. Y., Betz, S., Penzenstadler, B., Crick, T., & Brooks, I. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107316 · Cited by 59
Technostress among teachers: A systematic literature review and future research agenda
Yang, D., Liu, J., Wang, H., Chen, P., Wang, C., & Metwally, A. H. S. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108619 · Cited by 60
User Experience Guidelines for Designing HMD Extended Reality Applications
Vi, S., Silva, T. C., & Maurer, F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29390-1_18 · Cited by 78
“After a quick overview, using backward snowballing as a complement to SLRs [26], we selected resources which focused on providing information regarding UX design for different XR technologies.”
MethodA review of work-life balance in the construction industry
Tijani, B., Osei‐Kyei, R., & Feng, Y. (2020) · International Journal of Construction Management · doi:10.1080/15623599.2020.1819582 · Cited by 76
“WFC is defined as ‘a form of inter-role conflict, in which role pressures from the work and family domains are mutually incompatible in some respect’ (Greenhaus and Beutell 1985).”
BackgroundInfluentialOccupational Accidents Related to Heavy Machinery: A Systematic Review
Duarte, J., Marques, A. T., & Baptista, J. S. (2021) · Safety · doi:10.3390/safety7010021 · Cited by 58
“After analysing their references by title and abstract, and according to the snowballing technique procedures [39], 6 more papers were added to the study, which led to a final inclusion of 16 studies (Figure 1).”
BackgroundMethodWhy many challenges with GUI test automation (will) remain
Nass, M., Alégroth, E., & FELDT, R. H. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106625 · Cited by 90
Ghafoor, S., Hosseini, M. R., Kocatürk, T., Weiss, M., & Barnett, M. R. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.136845 · Cited by 69
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.