Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 1,501–1,600 of 3,932
Akresh, M. E., King, D. I., McInvale, S. L., Larkin, J. L., & D’Amato, A. W. (2023) · Ecosphere · doi:10.1002/ecs2.4315 · Cited by 20
“We also examined reference lists and used “ snowballing ” techniques (Wohlin, 2014).”
MethodA framework for designing interactive mobile training course content using augmented reality
Al-Amri, S., Hamid, S. B., Noor, N. F. B. M., & Gani, A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14561-4 · Cited by 35
“The second method was the snowballing technique, which included searching references and bibliographies utilizing main articles or books linked to the study topic [74].”
MethodDSS from an RE Perspective: A systematic mapping
García, S., Romero, O., & Raventós, R. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.03.046 · Cited by 15
“(2) Extended search (snowballing technique): to systematically search for related literature, we implemented the snowballing technique suggested in (Wohlin, 2014) by using the collected literature from the previous step and iteratively extending our results by revising the citations and references.”
MethodA Model-Based Approach to Systematic Review of Research Literature
Barat, S., Clark, T., Barn, B. S., & Kulkarni, V. (2017) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3021460.3021462 · Cited by 25
“Biasness and Threats to validity Study identification Finding publications using search string, snowballing [26], etc,.”
MethodModel-based testing areas, tools and challenges: A tertiary study
Villalobos-Arias, L., Quesada-López, C., Martinez, A., & Jenkins, M. (2019) · CLEI electronic journal · doi:10.19153/cleiej.22.1.3 · Cited by 22
“We conducted a tertiary study based on the guidelines described in [7, 8] as well as the recommendations stated in [9, 10, 11].”
MethodTowards the Description and Representation of Smartness in IoT Scenarios Specification
de Souza, B. P., Motta, R. C., & Travassos, G. H. (2019) · Brazilian Symposium on Software Engineering · doi:10.1145/3350768.3351797 · Cited by 18
“Snowballing is a search method for literature systematic studies, whose use is given through a reference list of works or citations [25].”
MethodAdoption of Software Testing in Internet of Things
Cortés, M. I., Saraiva, R., de Souza, M. D. C. O., et al. (2019) · doi:10.1145/3356317.3356326 · Cited by 14
“We have not performed snowballing [23], a useful technique that consists of checking the reference lists of the read studies, aiming at finding additional studies that were not retrieved in the automated search procedure.”
MethodQuality attributes use in architecture design decision methods: research and practice
Lytra, I., Carrillo, C., Capilla, R., et al. (2019) · Computing · doi:10.1007/s00607-019-00758-9 · Cited by 14
“For research question RQ1, we are interested in whether the QAs appear in the decisión making process, in the documentation of the design decisions, or both.”
BackgroundMethodDeep Learning Techniques for Stock Market Prediction in the European Union: A Systematic Review
Ketsetsis, A. P., Kourounis, C., Spanos, G., Giannoutakis, K. M., Pavlidis, P., Vazakidis, D., Champeris, T., Thomas, D., & Tzovaras, D. K. (2020) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci51800.2020.00107 · Cited by 15
“The search strategy used in the present review includes: the snowball technique [19], both backward and forward and the automated search in appropriate databases (Google Scholar and Connected Papers), as a complementary to the snowball technique.”
MethodResearch on Third-Party Libraries in AndroidApps: A Taxonomy and Systematic LiteratureReviewPreprint
Zhan, X., Liu, T., Fan, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.03787 · Cited by 14
“In accordance with international academic rules [44], [45], publications that are less than five pages in IEEE/ACM-like double-column format or less than seven pages in LNCS-”
BackgroundMethodDafonte, K. K., Weber, L., Chmielewski, F., Böhmer, A. M., Lutz, P. L., Hartmann, G., Jaehde, U., & Coenen, M. (2023) · Clinical Drug Investigation · doi:10.1007/s40261-023-01289-0 · Cited by 21
Tipping points in coupled human–environment system models: a review
Farahbakhsh, I., Bauch, C. T., & Anand, M. (2024) · Earth System Dynamics · doi:10.5194/esd-15-947-2024 · Cited by 25
Xu, W., & Ouyang, F. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100685 · Cited by 23
When Deep Learning Meets Information Retrieval-based Bug Localization: A Survey
Niu, F., Li, C., Liu, K., Xia, X., & Lo, D. (2025) · ACM Computing Surveys · doi:10.1145/3734217 · Cited by 28
“Quality assessment is a vital process of survey to ensure that we form a proper and fair representation of the research works [66].”
A Functional Software Reference Architecture for LLM-Integrated Systems
Bucaioni, A., Weyssow, M., He, J., Lyu, Y., & Lo, D. (2025) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c65153.2025.00006 · Cited by 29
“To mitigate biases [13], backward and forward snowballing [24] identified one additional study, resulting in three primary studies [14], [18], [19] 1 .”
Wawire, B. A., Kiambati, F., Muhia, N., Gathoni, G., & Abuya, B. A. (2025) · Journal of Curriculum Studies · doi:10.1080/00220272.2025.2523429 · Cited by 39
“In addition, the researchers employed the snowballing approach to identify further relevant articles that were included in the reference list (backward snowballing) or had cited the identified articles (forward snow-balling) (Wohlin, 2014).”
Bakogeorgou, S., Filippatos, C., Malandrakis, P., Tentolouris, A., Terpos, E., Gavriatopoulou, M., & Ntanasis‐Stathopoulos, I. (2025) · Cancers · doi:10.3390/cancers17172727 · Cited by 16
“Additionally, a comprehensive and systematic snowball approach was used to capture all relevant records by reviewing the reference lists of included studies, ensuring comprehensive coverage and minimizing the risk of omitting previously cited literature [21].”
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 51
“To complement the systematic search, we performed backward snowballing (Wohlin, 2014) on papers deemed eligible for inclusion.”
Cohort Studies in Software Engineering: A Vision of the Future
Saarimäki, N., Lenarduzzi, V., Vegas, S., Juristo, N., & Taibi, D. (2020) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3382494.3422160 · Cited by 13
“In addition, Wohlin [30] has written a guideline for replicating SLRs and applying the snowballing technique for finding papers.”
MethodHow good are my search strings? Reflections on using an existing review as a quasi-gold standard
Tran, H. K. V., Börstler, J., bin Ali, N., & Unterkalmsteiner, M. (2021) · e-Informatica Software Engineering Journal · doi:10.37190/e-inf220103 · Cited by 13
“Moreover, we draw inspiration from the snowball sampling guidelines for a good initial set to propose recommendations for arriving at a good QGS [28].”
MethodTrends for the DevOps Security. A Systematic Literature Review
Leppänen, T., Honkaranta, A., & Costin, A. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-11510-3_12 · Cited by 19
Geropoulos, G., Esagian, S. M., Skarentzos, K., Ziogas, I. A., Katsaros, I., Kosmidis, D., Tsoulfas, G., Lawrence, D. R., & Panagiotopoulos, N. (2022) · Asian Cardiovascular and Thoracic Annals · doi:10.1177/02184923221115970 · Cited by 18
“The reference lists and all previously published systematic reviews were thoroughly searched for missed studies eligible for inclusion based on the “snowball” methodology.(14)”
MethodE-Cardiac Care: A Comprehensive Systematic Literature Review
Umar, U., Nayab, S., Irfan, R., Khan, M. A., & Umer, A. (2022) · Sensors · doi:10.3390/s22208073 · Cited by 16
“The criteria used for the selection of search terms or keywords is mentioned below [16]:”
Solís, F. M., & Critikián, D. M. (2022) · Doxa Comunicación Revista interdisciplinar de estudios de comunicación y ciencias sociales · doi:10.31921/doxacom.n36a1707 · Cited by 32
Survival and cause-specific mortality of coyotes in Wisconsin
Margenau, L. L. S., Russell, R. E., Hanrahan, A. T., Roberts, N., Tack, J. L. P., & Storm, D. J. (2023) · Journal of Mammalogy · doi:10.1093/jmammal/gyad033 · Cited by 16
Yan, L., Winterbottom, D., & Liu, J. (2023) · Sustainability · doi:10.3390/su15076141 · Cited by 17
Identifying Industry 4.0 technologies enablers for knowledge management – a scoping review
Lista, A. P., Tortorella, G. L., Bouzon, M., Gao, S., & Chan, T. K. (2023) · The TQM Journal · doi:10.1108/tqm-05-2022-0173 · Cited by 15
Karlovšek, J., Meath, C., Miles-Mwangangi, L., MacDonald, C., & Brockmann, A. (2023) · Engineering Construction & Architectural Management · doi:10.1108/ecam-09-2022-0908 · Cited by 24
Identifying necessary conditions to deep-tech entrepreneurship
Dionísio, E. A., Júnior, E. I., Morini, C., & Quadros, R. (2023) · RAUSP Management Journal · doi:10.1108/rausp-09-2022-0203 · Cited by 23
A Systematic Review of Performance Measurement Systems and Their Relevance to Social Enterprises
Cosa, M., & Urban, B. (2023) · Journal of Social Entrepreneurship · doi:10.1080/19420676.2023.2236628 · Cited by 13
“Our search methodology concluded with a snowballing procedure (Wohlin 2014), where we searched references of primary studies and newer studies, citing them to find additional relevant research.”
Documentation Practices in Agile Software Development: A Systematic Literature Review
Islam, M. A., Hasan, R., & Eisty, N. U. (2023) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera57763.2023.10197828 · Cited by 24
“Finally, to satisfy the inclusion of relevant primary studies as much as possible, we performed backward and forward snowballing following the guideline provided by Wohlin [84] and included 20 more papers in our list of primary studies.”
MethodMunroe, A., & Dunleavy, M. (2023) · Irish Educational Studies · doi:10.1080/03323315.2023.2260371 · Cited by 19
“Additional articles were identi fi ed through snowballing (Wohlin 2014) and consultation with other researchers.”
BackgroundDefect Categorization in Compilers: A Multi-vocal Literature Review
Rahman, A., Bose, D. B., Barsha, F. L., & Pandita, R. (2023) · ACM Computing Surveys · doi:10.1145/3626313 · Cited by 14
“104:6 A. Rahman et al. To identify Internet artifacts and peer-reviewed publications, we use a set of search strings that were derived using snowballing technique [58] following the guidelines of Garousi et al. [22].”
MethodEnergy Prediction for Energy-Harvesting Wireless Sensor: A Systematic Mapping Study
Yuan, Z., Ge, Y., Wei, J., Yuan, S., Liu, R., & Mo, X. (2023) · Electronics · doi:10.3390/electronics12204304 · Cited by 18
“Furthermore, the snowball method [16] was employed to supplement the second round Furthermore, the snowball method [16] was employed to supplement the second round of screening, leading to the discovery of an additional 20 papers. of screening, leading to the discovery of an additional 20 papers.”
MethodMachine learning in Occupational Safety and Health - a systematic review
Maheronnaghsh, S., Zolfagharnasab, H., Gorgich, M., et al. (2023) · International Journal of Occupational and Environmental Safety · doi:10.24840/2184-0954_007-001_001586 · Cited by 13
Towards a Framework for Learning Content Analysis in K-12 AI/ML Education
Waite, J., Tshukudu, E., Cucuiat, V., Whyte, R. O., & Sentance, S. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343368 · Cited by 20
“Using these criteria, we “snow-balled” [19] further resources that were mentioned in the candidate reviews.”
MethodCorcoran, C. K., Cook, D., & Jóhannsdóttir, L. (2024) · Sustainable Production and Consumption · doi:10.1016/j.spc.2024.02.013 · Cited by 17
Security Issues in Special-Purpose Digital Radio Communication Systems: A Systematic Review
Dansarie, M. (2024) · IEEE Access · doi:10.1109/access.2024.3420091 · Cited by 18
“It should be diverse and cover papers from multiple publishers, years, and authors [6].”
BackgroundGeng, X., & Su, Y.-S. (2024) · International Journal of Science and Mathematics Education · doi:10.1007/s10763-024-10484-0 · Cited by 31
“After compiling all the studies, a snowball strategy recommended by Wohlin (2014) was employed to identify additional relevant studies by examining the reference lists in the electronic search papers.”
Comprehensive Insights into Artificial Intelligence for Dental Lesion Detection: A Systematic Review
Demir, K., SÖKMEN, Ö. Ç., Aksakalli, I. K., & AĞIRMAN, K. T. (2024) · Diagnostics · doi:10.3390/diagnostics14232768 · Cited by 26
Silva, I., Lapa, N., Ribeiro, H., & Duarte, E. (2025) · Applied Sciences · doi:10.3390/app15031037 · Cited by 13
“Throughout all the routes previously described, 15 additional articles were considered using the snowballing method, where reference lists are consulted to identify new possible articles [37].”
Herold, S., Blom, M., & Buckley, J. (2016) · European Conference on Software Architecture · doi:10.1145/2993412.3003396 · Cited by 21
“This risk could be further minimized by applying snowballing techniques [23] which were considered too time-consuming for this mapping study.”
BackgroundA Preliminary Systematic Investigation onto Sprayed Concrete's Environmental Performance
Saade, M. R. M., Passer, A., & Mittermayr, F. (2018) · Procedia CIRP · doi:10.1016/j.procir.2017.11.108 · Cited by 13
Guevara, J. N., Borys, D., DeAntonio, R., Guzmán-Holst, A., & Hoet, B. (2019) · Expert Review of Vaccines · doi:10.1080/14760584.2019.1688148 · Cited by 12
“Records retained after both screening steps were subsequently assessed for eligibility based on a full-text review by AGH during which the reference lists of the retrieved articles were also screened to identify additional records (‘snowballing’) [20].”
MethodSmart Grid Challenges Through the Lens of the European General Data Protection Regulation
Martinez, J., Ruíz, A., Puelles, J., et al. (2020) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-49644-9_7 · Cited by 13
“nologies area at Tecnalia, as well as legal experts from the KU Leuven Centre for IT & IP Law, along with a literature review using the snowballing approach [41].”
BackgroundAdding Sustainability to Lean Product Development
de Souza, J. P. E., & Dekkers, R. (2019) · Procedia Manufacturing · doi:10.1016/j.promfg.2020.01.325 · Cited by 17
What Developers (Care to) Recall: An Interview Survey on Smaller Systems
Krüger, J., & Hebig, R. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00015 · Cited by 13
“ask during their tasks, following established guidelines for SLRs [21], [65].”
BackgroundMethodMeta‐research publications in dentistry: a review
Faggion, C. M., Listl, S., & Smits, K. P. J. (2021) · European Journal Of Oral Sciences · doi:10.1111/eos.12748 · Cited by 14
Esagian, S. M., Spinos, D., Vasilopoulou, A., et al. (2021) · Journal of Nephrology · doi:10.1007/s40620-021-01016-y · Cited by 13
“We also manually screened the references of all included studies according to the “snowball” methodology [20].”
MethodA Systematic Literature Review on Teaching Programming to People with Cognitive Disabilities
de Araujo, E. C. J., & Andrade, W. L. (2021) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie49875.2021.9637361 · Cited by 14
A Comparative View of Agri-Environmental Indicators and Stakeholders’ Assessment of Their Quality
Spânu, I.-A., Ozunu, A., Petrescu, D. C., & Petrescu‐Mag, R. M. (2022) · Agriculture · doi:10.3390/agriculture12040490 · Cited by 15
The Role of Digital Infrastructure for the Industrialisation of Design for Additive Manufacturing
Mallalieu, A., Hajali, T., Isaksson, O., et al. (2022) · Proceedings of the Design Society · doi:10.1017/pds.2022.142 · Cited by 13
Managing Privacy in B2B Marketing: A Systematic Literature Review
Hill, S. R., & de Zubielqui, G. C. (2022) · Journal of Computer Information Systems · doi:10.1080/08874417.2022.2087792 · Cited by 19
Strengthening Schools’ Responses to Students’ Harmful Sexual Behaviors: A Scoping Review
Kor, K., Simpson, H., & Fabrianesi, B. (2023) · Trauma, Violence, & Abuse · doi:10.1177/15248380221111483 · Cited by 13
“The first and second authors read these publications in full and identified a further six publications through snowballing (Wohlin, 2014).”
BackgroundMotivating Factors to Self-Disclosure on Social Media: A Systematic Mapping
Zani, A. A. A., Norman, A. A., & Ghani, N. B. A. (2022) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2022.3184428 · Cited by 17
“Backward snowballing identifies additional papers by examining the reference list and citations in the studies being examined, as suggested by Wohlin [76].”
BackgroundMarois, A., & Lafond, D. (2022) · Cognition Technology & Work · doi:10.1007/s10111-022-00715-1 · Cited by 23
A Systematic Mapping on Discovery and Composition Mechanisms for Systems-of-Systems
Gomes, P., Cavalcante, E., Maia, P., Batista, T., & Oliveira, K. (2015) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2015.51 · Cited by 13
“In addition, we have not performed a snowballing [25], an useful technique that consists of checking the reference lists of the read studies aiming to find additional studies that were not retrieved in the automated search procedure.”
MethodA systematic literature mapping on the relationship between design patterns and bad smells
de Sousa, B. L., Bigonha, M. A. S., & Ferreira, K. A. M. (2018) · ACM Symposium on Applied Computing · doi:10.1145/3167132.3167295 · Cited by 13
“The analysis carried out in these studies indicate that when the design pattern implementation causes bad smells, consequently it degrades software quality, increasing complexity and damaging other important external attributes such as modularity, flexibility, testability, among others.”
BackgroundPast and future of software architectures for context-aware systems: A systematic mapping study
Roda, C., Navarro, E., Zdun, U., López‐Jaquero, V., & Simhandl, G. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.09.074 · Cited by 18
“those which appear in the references list (backward snowballing [53]).”
BackgroundMethodInfluentialGesell, C., Glas, A. H., & Eßig, M. (2018) · Journal of Business-to-Business Marketing · doi:10.1080/1051712x.2018.1532664 · Cited by 16
Global media flows: A qualitative review of research methods in audio-visual flow studies
Iordache, C., AUDENHOVE, L. V., & Loisen, J. (2018) · International Communication Gazette · doi:10.1177/1748048518808650 · Cited by 25
“Through both backward and forward snowballing methods (Wohlin, 2014), an increasing number of relevant flow studies were identified and added to the sample using studies that have been consistently referred to and acknowledged in specialized literature (e.”
InfluentialConstructing secure and memorable passwords
Kävrestad, J., Lennartsson, M., Birath, M., & Nohlberg, M. (2020) · Information and Computer Security · doi:10.1108/ics-07-2019-0077 · Cited by 20
URBANISATION AND THE CONCERNS FOR FOOD SECURITY IN MALAYSIA
Marzuki, A., & Jais, A. S. (2020) · PLANNING MALAYSIA · doi:10.21837/pm.v18i13.786 · Cited by 13
de Oliveira, D., Porto, F. A. M., Boeres, C., & de Oliveira, D. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5972 · Cited by 14
“Wefollowedtheguidelinesrecommended byWohlin 74 fordefiningtheseedsetofthesnowballingprocess(ie,weusedGoogleScholartogettheseedsetinsteadofchoosingaspecificrepos-itorylikeIEEEXPloreorACMDigitalLibrary).”
MethodProperty rights and the conservation of forests in communal areas in Zimbabwe
Mudombi-Rusinamhodzi, G., & Thiel, A. (2020) · Forest Policy and Economics · doi:10.1016/j.forpol.2020.102315 · Cited by 15
de Cerqueira, B. B., Barbosa, D. N. F., Mossmann, J. B., Cardoso, C. D. O., & Barbosa, J. L. V. (2020) · Applied Neuropsychology Child · doi:10.1080/21622965.2020.1843040 · Cited by 16
Modelling and Simulation in Industry 4.0
Ferreira, W. D. P., Armellini, F., de Santa-Eulália, L. A., & Rebolledo, C. (2021) · Studies in computational intelligence · doi:10.1007/978-3-030-61045-6_5 · Cited by 17
Educational use of Voice Assistants and Smart Speakers
Tsourakas, T., Terzopoulos, G., & Goumas, S. K. (2021) · Journal of Engineering Science and Technology Review · doi:10.25103/jestr.144.01 · Cited by 15
“In order to retrieve sufficient and high-quality papers, the snowball technique as described in [6] was used.”
MethodGilbert, G., Holdsworth, S., & Kyle, L. (2017) · VOLUNTAS International Journal of Voluntary and Nonprofit Organizations · doi:10.1007/s11266-016-9821-1 · Cited by 20
Semantic modeling for engineering data analytics solutions
Bandara, M., & Rabhi, F. (2019) · Semantic Web · doi:10.3233/sw-190352 · Cited by 14
“Findings were extended through snowballing approach proposed by Wohlin [19].”
BackgroundMethodInfluentialMachine Learning for Detecting Data Exfiltration: A ReviewPreprint
Sabir, B., Ullah, F., Babar, M. A., & Gaire, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09344 · Cited by 13
“It was then complemented with forward and backwards snowballing [55].”
MethodIntegrating security controls in DevSecOps: Challenges, solutions, and future research directions
Sinan, M., Shahin, M., & Gondal, I. (2025) · Journal of Software: Evolution and Process · doi:10.1002/smr.70029 · Cited by 13
de Chaves, S. A., & Benitti, F. B. V. (2025) · ACM Computing Surveys · doi:10.1145/3715903 · Cited by 21
“To complement this, we also employ snowballing—a technique recommended by Wohlin [ 30 ]—to uncover additional studies through references and citations.”
de Witt, M., Stefánsson, H., Valfells, Á., & Larsen, J. N. (2021) · Sustainability · doi:10.3390/su13168708 · Cited by 17
“The scoping literature review also provides a starting point for a more detailed literature review [34].”
BackgroundVanhala, E., Kasurinen, J., Knutas, A., & Herala, A. (2022) · IEEE Access · doi:10.1109/access.2022.3165079 · Cited by 18
“In the identification of relevant primary studies, the application level of strategies such as snowball sampling [43] was somewhat more positive than the application of a defined process for developing keywords or search strings: 80 primary studies (18.9”
BackgroundInfluentialThe Diffusion of Human Resource Transformation in Scandinavia: A Supply-Side Perspective
Madsen, D. Ø., & Slåtten, K. (2022) · Societies · doi:10.3390/soc12030081 · Cited by 15
Natora, A., Oxley, J. A., Barclay, L., Taylor, K., Bolam, B., & Haines, T. P. (2022) · International Journal of Public Health · doi:10.3389/ijph.2022.1604604 · Cited by 13
STATE OF GENDER EQUALITY IN AND BY ARTIFICIAL INTELLIGENCE
(2022) · IADIS INTERNATIONAL JOURNAL ON COMPUTER SCIENCE AND INFORMATION SYSTEMS · doi:10.33965/ijcsis_2022170203 · Cited by 20
“In the same way, the snowballing effect (Wohlin, 2014) will be followed, so the documents referenced in the considered studies will also be evaluated for their possible inclusion.”
BackgroundInformation overload research in accounting: a systematic review of the literature
Hartmann, M., & Weißenberger, B. E. (2023) · Management Review Quarterly · doi:10.1007/s11301-023-00343-7 · Cited by 27
“Still, we deviate from the procedure outlined in Wohlin (2014) in one aspect, as we include a paper if it is relevant, even if the original article in which it was cited, is later on excluded.”
InfluentialDornauer, B., & Felderer, M. (2023) · IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft) · doi:10.1109/mobilsoft59058.2023.00017 · Cited by 23
“Additionally, we employed guidelines by Wohlin [21] to specifically utilize forward and backward snowballing as a search method.”
MethodInfluentialCommit Classification Into Software Maintenance Activities: A Systematic Literature Review
Heričko, T., & Šumak, B. (2023) · IEEE Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00254 · Cited by 12
“Furthermore, we utilized the forward snowballing suggested by Wohlin [8] to supplement the automated search.”
MethodQuantifying Requirements Technical Debt: A Systematic Mapping Study and a Conceptual Model
Perera, J., Tempero, E. D., Tu, Y., & Blincoe, K. (2023) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re57278.2023.00021 · Cited by 14
“Forward and Backward reference snowballing [28] was performed on the final set of articles to avoid the possibility of missing relevant articles.”
MethodSelf-Adaptive Testing in the Field
da Silva, S. S., Pelliccione, P., & Bertolino, A. (2023) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3627163 · Cited by 21
A systematic mapping study on security for systems of systems
Olivero, M. Á., Bertolino, A., Mayo, F. J. D., Escalona, M. J., & Matteucci, I. (2023) · International Journal of Information Security · doi:10.1007/s10207-023-00757-0 · Cited by 15
“Snowballing backward and forward [19] was applied.”
Italia, M., Papile, F., Santi, R., & Curto, B. D. (2023) · Sustainability · doi:10.3390/su152416689 · Cited by 20
“From these results, other sources were obtained through snowballing sampling analysis [37] of previous and subsequent academic publications discussing sustainable selection of materials in product design.”
ResultAghakhani, G., Wautelet, Y., Dijkmans, D., Heng, S., Poelmans, S., & Kolp, M. (2024) · Enterprise Information Systems · doi:10.1080/17517575.2023.2300089 · Cited by 15
“To collect a comprehensive set of papers, we combined both a classical SLR (Kitchenham and Charters 2007) with a forward snowballing sampling (Wohlin 2014).”
MethodJacquet, L., Duigou, A. L., & Kerbrat, O. (2024) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-023-02269-4 · Cited by 12
“The so-called snowballing method (Wohlin 2014) was used to find articles that might otherwise have been disregarded.”
Mantau, M. J., & Benitti, F. B. V. (2024) · Universal Access in the Information Society · doi:10.1007/s10209-024-01110-5 · Cited by 15
“Based on the results obtained in step 2, we applied the backward and forward snowball-ing techniques, as presented by [103].”
Quantum Machine Learning in Intrusion Detection Systems: A Systematic Mapping Study
Faker, O., & Çağıltay, N. E. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-7886-1_9 · Cited by 15
An Analysis of MLOps Architectures: A Systematic Mapping Study
Najafabadi, F. A., Bogner, J., Gerostathopoulos, I., & Lago, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70797-1_5 · Cited by 28
“Step 4: Snowballing We apply backward and forward snowballing to enrich the results obtained via automated search, as suggested by Wohlin et al. [25].”
MethodCritical Review of Virtual Reality Applications in Offsite Construction Research
Assaf, M., Al‐Hussein, M., & Li, X. (2024) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-14837 · Cited by 15
When we talk about Big Data, What do we really mean? Toward a more precise definition of Big Data
Han, X., Gstrein, O. J., & Andrikopoulos, V. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1441869 · Cited by 20
“In addition to the papers collected, we employed the forward snowball sampling method (Wohlin, 2014) to supplement our database.”
MethodMatsimbe, J. J., Dinka, M. O., Olukanni, D. O., & Musonda, I. (2024) · Discover Applied Sciences · doi:10.1007/s42452-024-06244-y · Cited by 15
“The snowballing technique [46] and ROBIS tool [47, 48] were further adopted to collect additional data and assess the risk of bias, respectively.”
A Holistic Analysis on Risks of Post-Disaster Reconstruction Using RStudio Bibliometrix
Serter, M., & Ayalp, G. G. (2024) · Sustainability · doi:10.3390/su16219463 · Cited by 14
Collaborative Distributed Machine Learning
Jin, D. S., Kannengießer, N., Rank, S., & Sunyaev, A. (2024) · ACM Computing Surveys · doi:10.1145/3704807 · Cited by 23
“To find CDML systems for testing the external validity of the CDML system conceptualization, we applied a backward search and a forward search [47] to the set of relevant publications.”
Wawrzynkowski, P., Molins, C., & Lloret, J. (2024) · Marine Policy · doi:10.1016/j.marpol.2024.106518 · Cited by 19
Khalafi, M., Rosenkranz, S. K., Ghasemi, F., Kheradmand, S., Maleki, A. H., Korivi, M., & Tsao, J.-P. (2025) · Nutrition & Metabolism · doi:10.1186/s12986-024-00885-x · Cited by 17
“Furthermore, snowballing search through backward searching using article reference list and forward searching by looking at who cited it were also performed [57].”
Famiglietti, J., Acconito, L., Arpagaus, C., & Toppi, T. (2025) · Energy Conversion and Management X · doi:10.1016/j.ecmx.2025.100947 · Cited by 15
Software testing for extended reality applications: a systematic mapping study
Gu, R., Rojas, J. M., & Shin, D. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00523-7 · Cited by 23
SoK: Software Compartmentalization
Lefeuvre, H., Dautenhahn, N., Chisnall, D., & Olivier, P. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00075 · Cited by 41
“To cover works prior to 2003 and from other venues, we apply snowballing [250] to each paper of our set to identify 66 further works, and complete the list with 49 more from our knowledge of the industry and literature, totaling 211 works.”
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.