Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 201–300 of 3,934
A review of the use of artificial intelligence methods in infrastructure systems
McMillan, L., & Varga, L. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105472 · Cited by 164
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
“[14] Samireh Jalali and Claes Wohlin.”
MethodSoftware startup engineering: A systematic mapping study
Berg, V., Birkeland, J., Nguyen-Duc, A., Pappas, I. O., & Jaccheri, L. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.06.043 · Cited by 125
“using the forward snowballing technique [42], to identify additional papers not discovered by the search string.”
BackgroundMethodInfluentialA Systematic Review of Cloud Modeling Languages
Bergmayr, A., Breitenbücher, U., Ferry, N., Rossini, A., Solberg, A., Wimmer, M., Kappel, G., & Leymann, F. (2018) · ACM Computing Surveys · doi:10.1145/3150227 · Cited by 144
“In particular, we used backward snowballing [Wohlin 2014] with the set of relevant studies by using their reference lists to identify new papers to include in the study.”
MethodAndroid Mobile Malware Detection Using Machine Learning: A Systematic Review
Senanayake, J., Kalutarage, H., & Al-Kadri, M. O. (2021) · Electronics · doi:10.3390/electronics10131606 · Cited by 162
“We performed the snowballing process [33], considering all the references presented in the retrieved papers and evaluating all the papers referencing the retrieved ones, which resulted in two additional relevant paper.”
MethodThe effectiveness of gamification in programming education: Evidence from a meta-analysis
Zhan, Z., He, L., Tong, Y., Liang, X., Guo, S., & Lan, X. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100096 · Cited by 233
Rasheed, J., Jamil, A., Hameed, A. A., Aftab, U., Aftab, J., Shah, S. A., & Draheim, D. (2020) · Chaos, Solitons & Fractals · doi:10.1016/j.chaos.2020.110337 · Cited by 121
Software-testing education: A systematic literature mapping
Garousi, V., Rainer, A., Lauvås, P., & Arcuri, A. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110570 · Cited by 158
“11 Figure 4- A screenshot from the search activity using Google Scholar (directly- or potentially-relevant papers are highlighted by red boxes) We conducted forward and backward snowballing [48] on the set of papers already in the pool.”
BackgroundMethodInfluentialEngesser, V. K., Rombaut, E., Vanhaverbeke, L., & Lebeau, P. (2023) · Sustainability · doi:10.3390/su15032774 · Cited by 178
“Choosing appropriate search strings may be challenging, as terminology is not always applied in a standardized way, which can lead to less useful search results [23].”
BackgroundMethodInfluentialAn exploration of IoT platform development
Fahmideh, M., & Zowghi, D. (2019) · Information Systems · doi:10.1016/j.is.2019.06.005 · Cited by 138
“To avoid missing any related paper, conducting the literature review was followed by performing the reference snowballing technique (Wohlin, 2014) in the sense that studies cited in the references and related work section of the paper were feeding into the next run of the literature search.”
BackgroundMethodInfluentialAn Update on Effort Estimation in Agile Software Development: A Systematic Literature Review
Fernández-Diego, M., Mendez, E. R., González-Ladrón-de-Guevara, F., Abrahão, S., & Insfrán, E. (2020) · IEEE Access · doi:10.1109/access.2020.3021664 · Cited by 169
“This phase consists of using the bibliographic references found in the papers to identify additional studies related to the objective of the search [43].”
BackgroundFrom Bitcoin to Central Bank Digital Currencies: Making Sense of the Digital Money Revolution
da Cunha, P. R., Melo, P., & Sebastião, H. (2021) · Future Internet · doi:10.3390/fi13070165 · Cited by 119
“We have performed a snowballing review [21] of English language literature on this topic, encompassing both academic and grey sources (e.”
MethodA Survey on Concept Drift in Process Mining
Sato, D. M. V., de Freitas, S. C., Barddal, J. P., & Scalabrin, E. E. (2021) · ACM Computing Surveys · doi:10.1145/3472752 · Cited by 161
“Finally, we applied backward snowballing [80] while following the ICs and ECs, including three papers summing up a total of 45 papers.”
MethodAir Pollution Dispersion Modelling in Urban Environment Using CFD: A Systematic Review
Pantusheva, M., Mitkov, R., Hristov, P. O., & Petrova‐Antonova, D. (2022) · Atmosphere · doi:10.3390/atmos13101640 · Cited by 118
“A method referred to as a ‘bibliographic search’, ‘citation tracking’, ‘snowballing‘, or ‘pearl growing’ was applied as a complimentary technique to identify more papers relevant to the current study [43,44].”
MethodAligning software engineering education with industrial needs: A meta-analysis
Garousi, V., Giray, G., Tüzün, E., Catal, C., & Felderer, M. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.06.044 · Cited by 119
“As recommended by systematic review guidelines, we also conucted forward and backward snowballing ( Wohlin, 2014 ) on the apers already in the pool, to include all the relevant sources s much as possible.”
InfluentialMertala, P. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.08.003 · Cited by 190
Regenerative Agriculture and Its Potential to Improve Farmscape Function
O’Donoghue, T., Minasny, B., & McBratney, A. B. (2022) · Sustainability · doi:10.3390/su14105815 · Cited by 168
A Multidisciplinary Review of Recycling Methods for End-of-Life Wind Turbine Blades
Paulsen, E. B., & Enevoldsen, P. (2021) · Energies · doi:10.3390/en14144247 · Cited by 145
“Based on selected keywords, a few introductory articles are picked, forming the base for the snowballing effect [25].”
BackgroundFairness and Bias in Algorithmic Hiring: A Multidisciplinary Survey
Fabris, A., Baranowska, N., Dennis, M. J., Graus, D., Hacker, P., Saldivar, J., Zuiderveen Borgesius, F., & Biega, A. J. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3696457 · Cited by 117
A Systematic Review of Digital Behaviour Change Interventions for More Sustainable Food Consumption
Hedin, B., Katzeff, C., Eriksson, E., & Pargman, D. S. (2019) · Sustainability · doi:10.3390/su11092638 · Cited by 163
“After removing duplicates and applying the inclusion and exclusion criteria, we used backward and forward snowballing to (1) search through the references of all selected papers (backward snowballing) and (2) search for more recent papers that referred to any of the selected papers to find relevant papers that could b…”
A Systematic Literature Review of Test Case Prioritization Using Genetic Algorithms
Bajaj, A., & Sangwan, O. P. (2019) · IEEE Access · doi:10.1109/access.2019.2938260 · Cited by 122
“Secondary Search: We have used the snowballing procedure [35], [36] for conducting the secondary search.”
MethodA review of code smell mining techniques
Rasool, G., & Arshad, Z. (2015) · Journal of Software Evolution and Process · doi:10.1002/smr.1737 · Cited by 143
“The guidelines for snowballing are presented by Wohlin [33].”
BackgroundDissanayake, N., Jayatilaka, A., Zahedi, M., & Babar, M. A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106771 · Cited by 192
“We applied snowballing [28] to scan the references of the selected studies to find more potential studies.”
MethodFerko, E., Bucaioni, A., & Behnam, M. (2022) · IEEE Access · doi:10.1109/access.2022.3172964 · Cited by 147
“Staring from these studies, we carried out closed recursive snowballing activities [22] for reducing possible threats to construct validity.”
MethodSelf-Sovereign Identity: A Systematic Review, Mapping and Taxonomy
Schardong, F., & Custódio, R. F. (2022) · Sensors · doi:10.3390/s22155641 · Cited by 198
“The combined output was a set of fifty-nine works which formed the input set for both forward and backward snowballing [89].”
BackgroundIntelligent automation for sustainable tourism: a systematic review
Majid, G. M., Tussyadiah, I. P., Kim, Y. R., & Pal, A. (2023) · Journal of Sustainable Tourism · doi:10.1080/09669582.2023.2246681 · Cited by 166
“Furthermore, the snowballing approach identified 23 relevant articles not retrieved in the previous search (Wohlin, 2014).”
BackgroundChallenges for Static Analysis of Java Reflection - Literature Review and Empirical Study
Landman, D., Serebrenik, A., & Vinju, J. J. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.53 · Cited by 162
“Two commonly used literature review techniques are snowballing [25], [26] and Systematic Literature Review (SLR) [27].”
MethodLone, A. H., & Naaz, R. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2020.100360 · Cited by 129
Compagnoni, M. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.133101 · Cited by 154
Systematic Mapping of Process Mining Studies in Healthcare
Erdoğan, T. G., & Tarhan, A. (2018) · IEEE Access · doi:10.1109/access.2018.2831244 · Cited by 143
“• References of papers already included in the pool using the backward and forward snowballing methods [29], • ResearchGate web pages of researchers, • Three specific websites (processmining.”
MethodNext generation swale design for stormwater runoff treatment: A comprehensive approach
Ekka, S. A., Rujner, H., Leonhardt, G., Blecken, G., Viklander, M., & Hunt, W. F. (2020) · Journal of Environmental Management · doi:10.1016/j.jenvman.2020.111756 · Cited by 135
Industrial sustainability: Modelling drivers and mechanisms with barriers
Neri, A., Cagno, E., Sebastiano, G. D., & Trianni, A. (2018) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.05.140 · Cited by 131
“We also used snowball method, i.e. starting from a set of selected contributions, we searched for other relevant literature looking at references and citations of the initial set of selected contributions (Heckathorn and Cameron, 2017; Skolarus et al., 2017; Wohlin, 2014).”
BackgroundMethodThreat analysis of software systems: A systematic literature review
Tuma, K., Calikli, G., & Scandariato, R. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.06.073 · Cited by 111
“We have also performed a backward snowballing search method [16].”
MethodSpam Review Detection Techniques: A Systematic Literature Review
Hussain, N., Mirza, H. T., Rasool, G., Hussain, I., & Kaleem, M. (2019) · Applied Sciences · doi:10.3390/app9050987 · Cited by 135
“Furthermore, snowball tracking technique is used to check the references of the studies [12].”
MethodComparative whole building LCAs: How far are our expectations from the documented evidence?
Saade, M. R. M., Guest, G., & Amor, B. (2019) · Building and Environment · doi:10.1016/j.buildenv.2019.106449 · Cited by 131
Chalmeta, R., & Santos-deLeón, N. J. (2020) · Sustainability · doi:10.3390/su12104108 · Cited by 135
“The methodology we followed combines Rowley and Slack’s proposal [17] and the snowball method [18] for item selection.”
MethodAlshami, A., Ali, E., Elsayed, M., Eltoukhy, A. E. E., & Zayed, T. M. (2024) · IEEE Access · doi:10.1109/access.2024.3392573 · Cited by 144
“On the other hand, the forward snowballing approach encompassed examining the articles that have cited the included ones [48].”
BackgroundMethodA retrospective analysis of maritime cyber security incidents
Meland, P. H., Bernsmed, K., Wille, E., Rødseth, Ø. J., & Nesheim, D. A. (2021) · TransNav, the International Journal on Marine Navigation and Safety of Sea Transportation · doi:10.12716/1001.15.03.04 · Cited by 112
“Furthermore, we applied a snowballing technique [83], which means that we screened our sources’ sources, to locate additional relevant literature that did not show up in our initial searches.”
MethodForberger, S., Reisch, L. A., Kampfmann, T., & Zeeb, H. (2019) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-019-0844-z · Cited by 137
“Second, forward snowball sampling was applied to include studies based on papers citing the paper under examination [71].”
MethodCrane operations and planning in modular integrated construction: Mixed review of literature
Hussein, M., & Zayed, T. M. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103466 · Cited by 122
Wehn, U., Gharesifard, M., Ceccaroni, L., Joyce, H. M., Ajates, R., Woods, S. M., Bilbao, A., Parkinson, S. T., Gold, M. M., & Wheatland, J. A. T. (2021) · Sustainability Science · doi:10.1007/s11625-021-00959-2 · Cited by 151
“In addition to the literature mentioned above, 12 publications were identified by the authors via backward and forward snowballing (Wohlin 2014).”
BackgroundAIS-based maritime anomaly traffic detection: A review
Ribeiro, C., Paes, A., & de Oliveira, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120561 · Cited by 148
The Emergence and Rise of Industry 4.0 Viewed through the Lens of Management Fashion Theory
Madsen, D. Ø. (2019) · Administrative Sciences · doi:10.3390/admsci9030071 · Cited by 139
Datasets are not enough: Challenges in labeling network traffic
Guerra, J. L., Catania, C. A., & Veas, E. E. (2022) · Computers & Security · doi:10.1016/j.cose.2022.102810 · Cited by 156
Wildlife crime: a conceptual integration, literature review, and methodological critique
Kurland, J., Pires, S. F., McFann, S. C., & Moreto, W. D. (2017) · Crime Science · doi:10.1186/s40163-017-0066-0 · Cited by 136
“Snowballing, or reference and citation tracking, was used to find articles based on literature already familiar to the authors through existing knowledge, previous research, and personal contacts (Greenhalgh and Peacock 2005; Wohlin 2014).”
MethodMachine Learning Testing: Survey, Landscapes and HorizonsPreprint
Zhang, J. M., Harman, M., Ma, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10742 · Cited by 108
“To ensure a high coverage of ML testing related papers, we therefore also performed snowballing [74] on each of the related papers found by keyword searching.”
MethodEnvironmental sustainability of negative emissions technologies: A review
Jeswani, H. K., Saharudin, D. M., & Azapagic, A. (2022) · Sustainable Production and Consumption · doi:10.1016/j.spc.2022.06.028 · Cited by 133
Model-based security engineering for cyber-physical systems: A systematic mapping study
Nguyen, P. H., Ali, S., & Yue, T. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2016.11.004 · Cited by 107
Software test maturity assessment and test process improvement: A multivocal literature review
Garousi, V., Felderer, M., & Hacaloğlu, T. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.01.001 · Cited by 105
“To ensure including all the relevant sources as much as possible, we conducted forward and backward snowballing [52] , as recommended by systematic review guidelines, on the set of papers already in the pool.”
BackgroundMethodCrowdwork platform governance toward organizational value creation
Gol, E. S., Stein, M., & Avital, M. (2019) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2019.01.001 · Cited by 144
“Finally, when a number of relevant papers had been identified, we utilized a snowballing approach to make sure we had not missed an essential source (Wohlin, 2014).”
BackgroundStrategies to reduce food waste in the foodservices sector: A systematic review
Vizzoto, F., Testa, F., & Iraldo, F. (2021) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2021.102933 · Cited by 136
Exploring Explainability: A Definition, a Model, and a Knowledge Catalogue
Chazette, L., Brunotte, W., & Speith, T. (2021) · 2021 IEEE 29th International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00025 · Cited by 105
“A. Data Collection and Analysis 1) Systematic Literature Review: We followed guidelines from Kitchenham et al. [33], and Wohlin [34] when conducting our SLR.”
MethodAndersen, C. M. E., Rasmussen, F. N., Habert, G., & Birgisdóttir, H. (2021) · Frontiers in Built Environment · doi:10.3389/fbuil.2021.729096 · Cited by 129
“The literature was collected and analyzed following the protocol for Systematic Literature Review (SLR) as well as the ‘ snowball ’ approach (de Almeida Biolchini et al., 2007; Higgins and Green, 2008; Wohlin, 2014).”
MethodKnowledge brokering to bridge the research‐practice gap in education: Where are we now?
Rycroft‐Smith, L. (2022) · Review of Education · doi:10.1002/rev3.3341 · Cited by 107
Augmented reality in architecture and construction education: state of the field and opportunities
Hajirasouli, A., & Banihashemi, S. (2022) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-022-00343-9 · Cited by 178
“This stage of search further involved a reverse search technique, or as described by Wohlin (2014), snowballing technique.”
MethodArdebili, A. A., Zappatore, M. S., Ramadan, A. I. H. A., Longo, A., & Ficarella, A. (2024) · Energy Informatics · doi:10.1186/s42162-024-00385-5 · Cited by 192
Multi-view approaches for software and system modelling: a systematic literature review
Cicchetti, A., Ciccozzi, F., & Pierantonio, A. (2019) · Software & Systems Modeling · doi:10.1007/s10270-018-00713-w · Cited by 131
“For the forward snowballing,Google Scholarwas used to obtain those studies citing the current one [26].”
BackgroundMethodInfluentialA systematic literature review of techniques and metrics to reduce the cost of mutation testing
Pizzoleto, A. V., Ferrari, F. C., Offutt, J., Fernandes, L., & Ribeiro, M. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.07.100 · Cited by 113
“Wohlin [188] suggested snowballing should be an iterative process that ends only when no new studies are found.”
BackgroundMethodInfluentialHallstedt, S., Isaksson, O., & Öhrwall Rönnbäck, A. (2020) · Sustainability · doi:10.3390/su122310222 · Cited by 105
“, a study of “references to references” [46], added up to the total number of 31 articles in the review.”
BackgroundZhou, X., Van Brummelen, J., & Lin, P. (2020) · arXiv · doi:10.48550/arxiv.2009.10228 · Cited by 104
“We collected all existing work using the “snowballing” [70] and keyword-search approaches.”
MethodDesign Patterns for Smart Contracts in the Ethereum Ecosystem
Wöhrer, M., & Zdun, U. (2018) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/cybermatics_2018.2018.00255 · Cited by 132
“, citations and links were followed and reference lists were studied during a process called snowballing [7].”
BackgroundBujold, A., Roberge‐Maltais, I., Parent‐Rocheleau, X., Boasen, J., Sénécal, S., & Léger, P. (2023) · AI and Ethics · doi:10.1007/s43681-023-00325-1 · Cited by 276
Using Forward Snowballing to update Systematic Reviews in Software Engineering
Felizardo, K. R., Mendes, E., Kalinowski, M., Souza, É. F., & Vijaykumar, N. L. (2016) · Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962630 · Cited by 102
“Wohlin [23] carried out the comparison using the results from applying snowballing to the results from a previously conducted SLR that employed database searches.”
BackgroundMethodInfluentialLaparoscopic versus robotic major hepatectomy: a systematic review and meta-analysis
Ziogas, I. A., Giannis, D., Esagian, S. M., Economopoulos, K. P., Tohme, S. T., & Geller, D. A. (2020) · Surgical Endoscopy · doi:10.1007/s00464-020-08008-2 · Cited by 109
“We also hand-searched the reference lists and previously published systematic reviews for potentially relevant, missed studies according to the systematic “snowball” procedure guidelines [40].”
MethodKähkönen, T., Blomqvist, K., Gillespie, N., & Vanhala, M. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.03.019 · Cited by 88
Contributions of Industry 4.0 to supply chain resilience
Tortorella, G. L., Fogliatto, F. S., Gao, S., & Chan, T. (2021) · The International Journal of Logistics Management · doi:10.1108/ijlm-12-2020-0494 · Cited by 137
Analysing app reviews for software engineering: a systematic literature review
Dąbrowski, J., Letier, E., Perini, A., & Susi, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10065-7 · Cited by 186
“Finally, we completed the searching with a snowballing procedure following guidelines proposed by Wohlin (2014).”
MethodExploring LLMs Applications in Law: A Literature Review on Current Legal NLP Approaches
Siino, M., Falco, M., Croce, D., & Rosso, P. (2025) · IEEE Access · doi:10.1109/access.2025.3533217 · Cited by 232
“To identify relevant papers, we employed an adaptation of PRISMA guidelines, including a database search and complemented by the backward snowballing procedure [55], in order to reduce the risk of overlooking pertinent documents within the time frame selected.”
Münster, S., Georgi, C., Heijne, K., Klamert, K., Noennig, J. R., Pump, M., Stelzle, B., & van der Meer, H. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.08.102 · Cited by 136
“For compiling this overview the general snowballing guidelines [49] were used.”
MethodA focus area maturity model for software ecosystem governance
Jansen, S. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106219 · Cited by 123
“The practices were found by taking the literature studies of Manikas [2] and Alves et al. [12] as a starting point and snow-balling forward and backward [25] .”
MethodPearson, J., Jackson, G., & McNamara, K. E. (2021) · The Anthropocene Review · doi:10.1177/20530196211005482 · Cited by 154
“Using the snowballing technique, the reference lists of these 54 items were also checked and any relevant items were included in the review (Wohlin, 2014).”
MethodMaaß, L., Freye, M., Pan, C.-C., Dassow, H.-H., Niess, J., & Jahnel, T. (2022) · JMIR mHealth and uHealth · doi:10.2196/37980 · Cited by 101
“This is essential as they all have different search terms, leading to overseen but relevant papers in systematic literature searches with a specific search string [31-33].”
BackgroundMethodCollaborative Model-Driven Software Engineering: A Classification Framework and a Research Map
Franzago, M., Ruscio, D. D., Malavolta, I., & Muccini, H. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2755039 · Cited by 119
Managing architectural technical debt: A unified model and systematic literature review
Besker, T., Martini, A., & Bosch, J. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.09.025 · Cited by 135
Application Programming Interface (API) Research: A Review of the Past to Inform the Future
Ofoeda, J., Boateng, R., & Effah, J. K. (2019) · International Journal of Enterprise Information Systems · doi:10.4018/ijeis.2019070105 · Cited by 212
Systematic Literature Review of IoT Botnet DDOS Attacks and Evaluation of Detection Techniques
Gelgi, M., Guan, Y., Arunachala, S., Rao, M. S. S., & Dragoni, N. (2024) · Sensors · doi:10.3390/s24113571 · Cited by 176
“In this literature review, strategies are followed based on guidelines from Petersen [26] and Wohlin [27].”
MethodSoftware product lines and variability modeling: A tertiary study
Raatikainen, M., Tiihonen, J., & Männistö, T. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.12.027 · Cited by 114
“A recent trend in searches is to carry out snowballing after the introduction of the guidelines ( Wohlin, 2014 ).”
BackgroundMethodInfluentialTest Case Prioritization in Continuous Integration environments: A systematic mapping study
Lima, J. P., & Vergílio, S. R. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106268 · Cited by 120
Yadav, G., Mangla, S. K., Bhattacharya, A., & Luthra, S. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.124186 · Cited by 111
“While the initial shortlisting of articles is performed according to the above procedure, the final scrutiny of the articles is conducted through forward snowball and backward snowball approach (Wohlin, 2014) which left us with 63 articles.”
MethodA Systematic Review on Design Thinking Integrated Learning in K-12 Education
Li, T., & Zhan, Z. (2022) · Applied Sciences · doi:10.3390/app12168077 · Cited by 188
“After the initial search, we utilized a snowball method [25] using the references in the selected articles before 14 March 2022 to search more literature.”
MethodInfluentialXin, R., Chen, P., & Zhao, Z. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111724 · Cited by 99
“We use the snowballing technique [36] to explore CauseInfer’s citation papers to identify valuable papers and establish a comprehensive understanding of related research in CI-based root cause localization.”
MethodSleep assessment using EEG-based wearables – A systematic review
de Gans, C. J., Burger, P. C., van den Ende, E. S., Hermanides, J., Nanayakkara, P. W. B., Gemke, R. J. B. J., Rutters, F., & Stenvers, D. J. (2024) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2024.101951 · Cited by 129
Enterprise maturity models: a systematic literature review
Neto, J. B. S. D. S., & Costa, A. P. C. S. (2019) · Enterprise Information Systems · doi:10.1080/17517575.2019.1575986 · Cited by 129
“Step (8) was performed according to the backward snowballing method described by Wohlin (2014). The objective of this procedure is to extract new documents not identified by the search string for inclusion in the final set of relevant articles.”
MethodEnforcing security in Internet of Things frameworks: A Systematic Literature Review
Aly, M., Khomh, F., Haoues, M., Quintero, A., & Yacout, S. (2019) · Internet of Things · doi:10.1016/j.iot.2019.100050 · Cited by 98
Liu, Y., Orsi, R. H., Gaballa, A., Wiedmann, M., Boor, K. J., & Guariglia-Oropeza, V. (2019) · Future Microbiology · doi:10.2217/fmb-2019-0072 · Cited by 106
Time pressure in software engineering: A systematic review
Kuutila, M., Mäntylä, M., Farooq, U., & Claes, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106257 · Cited by 153
“Wohlin’s snowballing guidelines [142] were used to construct a set of seed papers and snowball through them.”
MethodBlockchain Application for Central Banks: A Systematic Mapping Study
Dashkevich, N., Counsell, S., & Destefanis, G. (2020) · IEEE Access · doi:10.1109/access.2020.3012295 · Cited by 137
“We also performed a forward snowballing sampling technique on the most cited papers [16].”
BackgroundMethodInfluentialEcosystems in support of social entrepreneurs: a literature review
Gonzalez, A. A. D., & Dentchev, N. A. (2021) · Social enterprise journal · doi:10.1108/sej-08-2020-0064 · Cited by 122
Software architectures for robotic systems: A systematic mapping study
Ahmad, A., & Babar, M. A. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.08.039 · Cited by 118
“While this search strategy and search string composition significantly increases the search work [45], however, it enabled us to find a comprehensive set of the relevant study.”
BackgroundMethodInfluentialFactors influencing the understandability of process models: A systematic literature review
Dikici, A., Türetken, O., & Demirörs, O. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.09.001 · Cited by 120
“We also analyzed the references of these publications and identified 5 additional studies following a snowballing approach proposed in [28] that were missing in our initial master list.”
BackgroundStrategies to Improve the Energy Performance of Buildings: A Review of Their Life Cycle Impact
Mirabella, N., Röck, M., Saade, M. R. M., Spirinckx, C., BOSMANS, M., Allacker, K., & Passer, A. (2018) · Buildings · doi:10.3390/buildings8080105 · Cited by 113
Maniriho, P., Mahmood, A. N., & Chowdhury, M. J. M. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.11.030 · Cited by 120
Algorithmic discrimination in the credit domain: what do we know about it?
García, A. C. B., García, M., & C, R. R. B. (2023) · AI & Society · doi:10.1007/s00146-023-01676-3 · Cited by 174
“…as a conference paper, journal article, book chapter, or technical report, • I4: is published between 2017 and 2022, Additional papers were included as a result of finding relevant material from citations in the original list of retrieved papers, in a process called snowballing (Wohlin 2014).”
A systematic mapping study of search-based software engineering for software product lines
Lopez-Herrejon, R. E., Linsbauer, L., & Egyed, A. F. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.01.008 · Cited by 101
Schulte, P. A., Streit, J. M. K., Sheriff, F., Delclos, G. L., Felknor, S. A., Tamers, S. L., Fendinger, S., Grosch, J. W., & Sala, R. M. (2020) · Annals of Work Exposures and Health · doi:10.1093/annweh/wxaa051 · Cited by 130
“The searches were completed using backward and forward snowball approaches (Wohlin, 2014) as well as ad hoc searches of relevant literature to December 2019.”
MethodHow to measure light pollution—A systematic review of methods and applications
Mander, S., Alam, F., Lovreglio, R., & Ooi, M. P. (2023) · Sustainable Cities and Society · doi:10.1016/j.scs.2023.104465 · Cited by 124
Arthurs, L. A., & Kreager, B. Z. (2017) · International Journal of Science Education · doi:10.1080/09500693.2017.1363925 · Cited by 95
Health-related fake news on social media platforms: A systematic literature review
Melchior, C., & Oliveira, M. F. D. A. (2021) · New Media & Society · doi:10.1177/14614448211038762 · Cited by 125
“…peer review, focused on analyzing health-related fake news (including misinformation and disinformation) shared on SMPs. Subsequently, in the third step, we maximized the scope of the search by performing a single iteration of the snowballing technique (Wohlin, 2014) using the 179 articles found.”
Teleportation in Virtual Reality; A Mini-Review
Prithul, A., Adhanom, I. B., & Folmer, E. (2021) · Frontiers in Virtual Reality · doi:10.3389/frvir.2021.730792 · Cited by 143
“The snowballing search method identifies a set of initial papers and uses the reference list of these papers (backward snowballing) or the citations to these papers (forward snowballing) to search for relevant studies (Wohlin, 2014).”
MethodInfluentialData cleaning and machine learning: a systematic literature review
Côté, P.-O., Nikanjam, A., Ahmed, N., Humeniuk, D. V., & Khomh, F. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00453-w · Cited by 163
“Snowballing refers to the act of discovering new papers through paper references (Wohlin, 2014).”
BackgroundInfluential
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.