Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 601–700 of 3,934
Key performance indicators for business models: a systematic review and catalog
van de Ven, M., Machado, P. L., Athanasopoulou, A., Aysolmaz, B., & Türetken, O. (2023) · Information Systems and e-Business Management · doi:10.1007/s10257-023-00650-2 · Cited by 119
“Despite following a rigorous research approach (Webster and Watson 2002; Wohlin 2014; Okoli 2015), our study is subject to some limitations that should be taken into consideration.”
BackgroundMethodInfluentialTrevisan, A. H., Boscarato, A., Acerbi, F., Terzi, S., & Sassanelli, C. (2025) · Journal of Environmental Management · doi:10.1016/j.jenvman.2025.124982 · Cited by 81
Quality characteristics and measures for human–computer interaction evaluation in ubiquitous systems
Carvalho, R. M., Andrade, R. M. C., de Oliveira, K. M., Santos, I. D. S., & Bezerra, C. I. M. (2016) · Software Quality Journal · doi:10.1007/s11219-016-9320-z · Cited by 55
“This procedure is adapted from the snowballing process proposed by Wohlin (2014).”
BackgroundMethodInfluentialA Systematic Mapping Study on Security in Agile Requirements Engineering
Villamizar, H., Kalinowski, M., Viana, M. L., & Méndez, D. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00080 · Cited by 59
“To address this threat, we used a hybrid search strategy that complements database search (on Scopus) with iterative backward and forward snowballing (using Google Scholar), which resulted in analyzing an extensive list of candidate publications to be included.”
MethodInfluentialEmpirical research for software architecture decision making: An analysis
Razavian, M., Paech, B., & Tang, A. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.12.003 · Cited by 69
“The legitimate finds from forward snowballing in Wohlin’s report represents a very small percentage [27].”
BackgroundMethodSQL Education: A Systematic Mapping Study and Future Research Agenda
Taipalus, T., & Seppänen, V. (2020) · ACM Transactions on Computing Education · doi:10.1145/3398377 · Cited by 69
“, following reference lists to find relevant studies) [114] twice.”
BackgroundSoftware Product Quality Metrics: A Systematic Mapping Study
Colakoglu, F. N., Yazıcı, A., & Mishra, A. (2021) · IEEE Access · doi:10.1109/access.2021.3054730 · Cited by 80
“The backward and forward snowballing technique [39] was utilized to eliminate the risk of missing relevant papers.”
MethodInteractive features of E-texts’ effects on learning: a systematic review and meta-analysis
Clinton-Lisell, V., Seipel, B., Gilpin, S., & Litzinger, C. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1943453 · Cited by 48
“A forwards search of work that had cited each of these relevant reports was conducted using Google Scholar (this search engine was chosen to avoid possible publisher bias, Wohlin, 2014), which yielded seven more reports.”
MethodBinkhonain, M., & Zhao, L. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.02.031 · Cited by 52
REVIEW ARTICLE: DUKUNGAN SOSIAL DALAM SITUASI PANDEMI COVID 19
Santoso, M. D. Y. (2020) · Jurnal Litbang Sukowati Media Penelitian dan Pengembangan · doi:10.32630/sukowati.v5i1.184 · Cited by 55
Live Coding: A Review of the Literature
Selvaraj, A., Zhang, E., Porter, L., & Soosai Raj, A. G. (2021) · Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3430665.3456382 · Cited by 47
“Snowballing is an efficient technique to identify important papers on a specific research area using reference lists and citations [49].”
MethodWohlin, C., & Runeson, P. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106678 · Cited by 95
Self-citationData quality issues in software fault prediction: a systematic literature review
Bhandari, K., Kumar, K., & Sangal, A. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10371-6 · Cited by 67
“Finally, to find the missing relevant studies from the literature, snowball sampling (Wohlin 2014) (also used in Azeem et al.”
BackgroundInfluentialA Systematic Literature Review of Autonomous and Connected Vehicles in Traffic Management
Alanazi, F. (2023) · Applied Sciences · doi:10.3390/app13031789 · Cited by 64
Baldassarre, B., Calabretta, G., Karpen, I. O., Bocken, N. M. P., & Hultink, E. J. (2024) · Journal of Business Ethics · doi:10.1007/s10551-023-05600-z · Cited by 88
“The sample was enlarged using the reference lists of these texts with a snowballing technique (Wohlin, 2014).”
MethodWhat software reuse benefits have been transferred to the industry? A systematic mapping study
Barros‐Justo, J. L., Pinciroli, F., Matalonga, S., & Martínez-Araujo, N. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.06.003 · Cited by 57
“Using the previous selected six works as an initial set, we performed backwards and forward snowballing [14], which add four more works to our list of articles.”
MethodInfluentialSoftware Testing in Introductory Programming Courses
Scatalon, L. P., Carver, J. C., Garcia, R. E., & Barbosa, E. F. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287384 · Cited by 49
“We conducted the search for relevant papers in two steps: (i) an automatic search in databases, which provided a list of relevant papers and (ii) a backward snowballing from this preliminary list to identify additional relevant papers [15].”
MethodA Survey on Collecting, Managing, and Analyzing Provenance from Scripts
Pimentel, J. F., Freire, J., Murta, L., & Braganholo, V. P. (2019) · ACM Computing Surveys · doi:10.1145/3311955 · Cited by 49
“Thus, instead of presenting the whole snowballing process in Figure 8, we present only what it would be if we had performed the whole snowballing on March 6th, 2017, with big backward and forward iterations, as described by Wohlin [128].”
MethodInfluentialSadoughi, F., Ali, O., & Erfannia, L. (2019) · Health Informatics Journal · doi:10.1177/1460458219879340 · Cited by 59
“46,47 The mix of the two strategies increases the likelihood that the systematic review covers many articles in any research area.”
BackgroundIntersecting Imaginaries: Visions of Decentralized Autonomous Systems
Lustig, C. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359312 · Cited by 47
“Intersecting Imaginaries: Visions of Decentralized Autonomous Systems 210:7 I gathered my literature through snowballing sampling [122], with a starting set of works on the politics and governance of protocols (i.e., [31, 32, 44, 79]) based on recommendations from colleagues, as my work had…”
BackgroundMethodFactors Determining City Brand Equity—A Systematic Literature Review
Górska-Warsewicz, H. (2020) · Sustainability · doi:10.3390/su12197858 · Cited by 64
“The snowball method was also used in two variants: as a backward snowball method consisting of checking reference lists in published articles and a forward snowball method consisting of identifying new tests quoting articles in SLR [74].”
MethodO&M Indoor Virtual Environments for People Who Are Blind: A Systematic Literature Review
Façanha, A. R., Darin, T., Viana, W., & Sánchez, J. (2020) · ACM Transactions on Accessible Computing · doi:10.1145/3395769 · Cited by 56
“After, the method has two distinct approaches: the Forward Snowballing, which searches for studies that cite the papers of the initial set; and the Backward Snowballing, which verifies new works coming from the list of references of the initial collection of documents [109].”
MethodInfluentialSoftware development with feature toggles: practices used by practitioners
Mahdavi-Hezaveh, R., Dremann, J., & Williams, L. A. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09901-z · Cited by 65
“In the relevant grey literature artifacts, we used a snowballing approach (Wohlin 2014).”
MethodThe Impact of Shipping on Air Quality in the Port Cities of the Mediterranean Area: A Review
Toscano, D. (2023) · Atmosphere · doi:10.3390/atmos14071180 · Cited by 62
“A complementary technique using “bibliographic search”, “citation tracking”, “snowballing”, or “pearl growing” was applied to identify more papers relevant to the current study [75,76].”
A systematic review of federated learning from clients’ perspective: challenges and solutions
Shanmugarasa, Y., Paik, H.-Y., Kanhere, S. S., & Zhu, L. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10563-8 · Cited by 67
Artificial Intelligence Applied to Software Testing: A Tertiary Study
Amalfitano, D., Faralli, S., Hauck, J. C. R., Matalonga, S., & Distante, D. (2023) · ACM Computing Surveys · doi:10.1145/3616372 · Cited by 76
“This stage refers to a snowballing process [104] that was conducted as a complementary search strategy to mitigate the threat of missing literature.”
BackgroundLean Six Sigma 4.0 – a framework and review for Lean Six Sigma practices in the digital era
Citybabu, G., & Yamini, S. (2023) · Benchmarking An International Journal · doi:10.1108/bij-09-2022-0586 · Cited by 62
Youth participation in cultural heritage management: a conceptual framework
Zhang, Y., Kaya, D. I., van Wesemael, P. J. V., & Colenbrander, B. (2023) · International Journal of Heritage Studies · doi:10.1080/13527258.2023.2275261 · Cited by 90
“After that, the ten most-relevant papers were selected through a snowballing procedure to collect other highly relevant sources of literature (Wohlin 2014).”
BackgroundMethodA Comprehensive Review of the Key Deterioration Factors of Concrete Bridge Decks
Ibrahim, A., Abdelkhalek, S., Zayed, T. M., Qureshi, A. H., & Abdelkader, E. M. (2024) · Buildings · doi:10.3390/buildings14113425 · Cited by 46
Search. Review. Repeat? An empirical study of threats to replicating SLR searches
Krüger, J., Lausberger, C., von Nostitz-Wallwitz, I., Saake, G., & Leich, T. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09763-0 · Cited by 69
“Essential for this search method is the initial set of papers that should be based on different communities, publishers, years, authors, and not too small (Wohlin 2014).”
Mobile app privacy in software engineering research: A systematic mapping study
Ebrahimi, F., Tushev, M., & Mahmoud, A. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106466 · Cited by 60
“To reduce the risk of omitting relevant studies, we also performed a lightweight backward-snowballing on the included papers [16].”
MethodPractical relevance of software engineering research: synthesizing the community’s voice
Garousi, V., Borg, M., & Oivo, M. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09803-0 · Cited by 78
“To ensure including all the relevant sources as much as possible, we also conducted forward and backward snowballing (Wohlin 2014) for the sources in formal literature, as recommended by systematic review guidelines, on the set of papers already in the pool.”
BackgroundMethodHotel Brand Loyalty—A Systematic Literature Review
Górska-Warsewicz, H., & Kulykovets, O. (2020) · Sustainability · doi:10.3390/su12124810 · Cited by 47
“The forward snowballing entailed identifying new studies citing the papers being examined in the systematic review [84].”
BackgroundDiks, M. E., Hiligsmann, M., & van der Putten, I. M. (2021) · BMC Infectious Diseases · doi:10.1186/s12879-021-06398-9 · Cited by 52
“Only two additional studies were identi ed, con rming the accuracy of key words used and suggesting that the primary was all-encompassing [27].”
BackgroundMethodEsagian, S. M., Kakos, C. D., Giorgakis, E., Burdine, L. J., Barreto, J. C., & Mavros, M. N. (2021) · Cancers · doi:10.3390/cancers13122984 · Cited by 49
“We also manually searched the references of all included articles and previously published narrative and systematic reviews for any that were missed but otherwise eligible according to the systematic “snowball” methodology [21].”
MethodA review of LED lamp recycling process from the 10 R strategy perspective
Rahman, S. M., Pompidou, S., Alix, T., & Laratte, B. (2021) · Sustainable Production and Consumption · doi:10.1016/j.spc.2021.07.025 · Cited by 50
Recommender systems in model-driven engineering: A systematic mapping review
Almonte, L., Guerra, E., Cantador, I., & de Lara, J. (2021) · Software & Systems Modeling · doi:10.1007/s10270-021-00905-x · Cited by 63
“Tomitigate this risk, we performed a final process of snowballing [138] to consider further relevant papers not included in the query results.”
MethodRespite for SMEs: A Systematic Review of Socio-Technical Cybersecurity Metrics
van Haastrecht, M., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021) · Applied Sciences · doi:10.3390/app11156909 · Cited by 45
“From a set of included papers, a researcher can find additional relevant papers by consulting references (backward snowballing) and citations (forward snowballing) [63].”
BackgroundMethodDesign Aspects in Repairability Scoring Systems: Comparing Their Objectivity and Completeness
Dangal, S., Faludi, J., & Balkenende, R. (2022) · Sustainability · doi:10.3390/su14148634 · Cited by 60
“of a paper or its citations to identify additional articles [16].”
Patellofemoral Pain Syndrome Risk Associated with Squats: A Systematic Review
Pereira, P. M., Baptista, J. S., da Conceição, F. A., Duarte, J., Ferraz, J., & Costa, J. T. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159241 · Cited by 53
“In the third step, articles and other works published before 2014 were considered in the review through the snowball procedure [31].”
BackgroundGraphQL: A Systematic Mapping Study
Quiña‐Mera, A., Fernández, P., García, J. M., & Ruiz-Cortés, A. (2023) · ACM Computing Surveys · doi:10.1145/3561818 · Cited by 90
“(3) To improve the quality of the primary study dataset, we initially piloted the snowballing search strategy [58] but found no new publications that met the inclusion criteria of the study filtering.”
BackgroundMethodIdentifying Leadership Competencies for Construction 4.0
Yang, K., Sunindijo, R. Y., & Wang, C. (2022) · Buildings · doi:10.3390/buildings12091434 · Cited by 65
Westaway, S., Grange, I., Smith, J., et al. (2022) · Land Use Policy · doi:10.1016/j.landusepol.2022.106502 · Cited by 45
Software engineering research on the Robot Operating System: A systematic mapping study
Albonico, M., Đorđević, M., Hamer, E., & Malavolta, I. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111574 · Cited by 63
A Systematic Literature Review on the Code Smells Datasets and Validation Mechanisms
Nasrabadi, M. Z., Parsa, S. A., Esmaili, E., & Palomba, F. (2023) · ACM Computing Surveys · doi:10.1145/3596908 · Cited by 61
Moriano, P., Hespeler, S. C., Li, M., & Mahbub, M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11292-w · Cited by 91
“In addition to electronic searches within these databases, supplementary strategies (forward/backward search [128]) were applied to identify additional key studies.”
Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning
Standen, M., Kim, J., & Szabo, C. (2025) · ACM Computing Surveys · doi:10.1145/3708320 · Cited by 90
“We used two search strings to find papers to address our research questions which were: — Finally, we used several well known papers that address our research questions [8, 120, 131] and used a forward snowball search method [139] to find additional relevant papers.”
The many faces of data-centric workflow optimization: a survey
Kougka, G., Gounaris, A., & Simitsis, A. (2018) · International Journal of Data Science and Analytics · doi:10.1007/s41060-018-0107-0 · Cited by 45
A review and analysis of the investment in, and cost structure of, intermodal rail terminals
Wiegmans, B., & Behdani, B. (2017) · Transport Reviews · doi:10.1080/01441647.2017.1297867 · Cited by 45
“Furthermore, the snowballing method (Wohlin, 2014) was employed (by looking for references in recent publications and citation to old relevant papers) and 13 more publications were added to the list, forming the list of 31 publications in total.”
BackgroundMethodSafety-Critical Systems and Agile Development: A Mapping Study
Kasauli, R., Knauss, E., Kanagwa, B., Nilsson, A., & Çalıklı, G. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00082 · Cited by 48
“Then, the first two authors went through the full texts of these remaining 69 papers collaboratively and selected 33 papers based on the inclusion and exclusion criteria in Table I, to which we added one more paper identified through snowballing [14] for consideration (indicated with +1 in Table I), resulting in a tot…”
BackgroundAlsanoosy, T., Spichkova, M., & Harland, J. (2019) · Requirements Engineering · doi:10.1007/s00766-019-00326-9 · Cited by 51
“Also, we followed the guidelines proposed by Wohlin [80] for conducting snowballing, and Zhang’s recommendations to identify relevant studies in software engineering [81].”
Methodde Campos, E. A. R., Tavana, M., ten Caten, C. S., Bouzon, M., & de Paula, I. C. (2020) · Environmental Science and Pollution Research · doi:10.1007/s11356-020-11138-8 · Cited by 56
Developing K-12 students’ programming ability: A systematic literature review
Sun, L., Guo, Z., & Zhou, D. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-10891-2 · Cited by 66
“The initial collection of articles plays a significant role in the subsequent snow-balling iterative process (Wohlin, 2014).”
BackgroundMethodInfluentialA Systematic Review of Recommender Systems and Their Applications in Cybersecurity
Pawlicka, A., Pawlicki, M., Kozik, R., & Choraś, R. S. (2021) · Sensors · doi:10.3390/s21155248 · Cited by 57
“The literature items were gathered, analyzed and selected with a mix of bibliographic methodologies, including the snowballing method [20], pearl-growing [21] and citation searching and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), all adjusted to fit the criteria of the PRISMA statement…”
MethodInnovation potential of megaprojects: a systematic literature review
Cantarelli, C. C., & Genovese, A. (2021) · Production Planning & Control · doi:10.1080/09537287.2021.2011462 · Cited by 52
“However, in order to ensure all relevant papers have been identified, we extended the selection of papers by adopting a snowballing procedure (backward search and forward search of retrieved papers) as a complementary search (Wohlin 2014; Jalali and Wohlin 2012).”
MethodAgnusdei, G. P., Coluccia, B., Elia, V., & Miglietta, P. P. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2022.01.312 · Cited by 47
Fahlstedt, O., Salaj, A. T., Lohne, J., & Bohne, R. A. (2022) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2022.112636 · Cited by 49
Designing for Care Ecosystems: a Literature Review of Technologies for Children with ADHD
Stefanidi, E., Schöning, J., Feger, S. S., Marshall, P., Rogers, Y., & Niess, J. (2022) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3501712.3529746 · Cited by 55
“Upon scanning the reference lists of these papers, another three entries were added, following Wohlin’s guidelines for snowballing in systematic reviews [73], resulting in a final corpus of 27 papers.”
MethodRichter, J. L., Svensson‐Hoglund, S., Dalhammar, C., Russell, J. D., & Thidell, Å. (2023) · Journal of Industrial Ecology · doi:10.1111/jiec.13315 · Cited by 44
“Another 11 articles were added through “snowballing” (see Wohlin, 2014) based on references within the reviewed articles and articles citing the reviewed articles, resulting in a total of 48 full articles reviewed and analyzed.”
BackgroundMobility trajectory generation: a survey
Kong, X., Chen, Q., Hou, M., Wang, H., & Xia, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10598-x · Cited by 66
“In the initial stage of the study, in accordance with the recommendations by Wohlin (2014), we utilized Google Scholar to conduct a literature search by employing diverse keywords, thereby mitigating potential publisher bias.”
MethodA Survey of Trust Management for Internet of Things
Konsta, A.-M., Lafuente, A. L., & Dragoni, N. (2023) · IEEE Access · doi:10.1109/access.2023.3327335 · Cited by 60
“After applying the above exclusion criteria, we also performed the snowballing [77] technique and ended up with 53 papers, which constitute the final pool.”
MethodA systematic review on how educators teach AI in K-12 education
Liu, X., & Zhong, B. (2024) · Educational Research Review · doi:10.1016/j.edurev.2024.100642 · Cited by 113
Schneider, M., Velasco‐Hernández, J. X., Min, K., Galan, D. I., Baca-Carrasco, D., Gompper, M. E., Hartskeerl, R. A., & Muñoz‐Zanzi, C. (2017) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph14060594 · Cited by 50
“Snowballing refers to reviewing papers of interest and using the references in, or citations to, the paper to identify additional relevant papers [40].”
Task Scheduling in Big Data Platforms: A Systematic Literature Review
Soualhia, M., Khomh, F., & Tahar, S. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.09.001 · Cited by 55
Architectural technical debt identification: the research landscape
Verdecchia, R., Malavolta, I., & Lago, P. (2018) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3194164.3194176 · Cited by 79
“[28], the adoption of such indexer results to constitute a sound choice to identify the initial set of literature for snowballing processes, (iii) the query results could be automatically extracted from the indexer.”
BackgroundBougioukas, K. I., Bouras, E., Avgerinos, K. I., Dardavessis, T., & Haidich, A. (2020) · Health Information & Libraries Journal · doi:10.1111/hir.12318 · Cited by 57
Video Game Automated Testing Approaches: An Assessment Framework
Albaghajati, A., & Ahmed, M. (2020) · IEEE Transactions on Games · doi:10.1109/tg.2020.3032796 · Cited by 70
“Backward snow-balling is the process of going through the references list of a study and identifying new papers to include from it [27].”
BackgroundA systematic mapping study on architectural smells detection
Mumtaz, H., Singh, P., & Blincoe, K. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110885 · Cited by 44
“3.4 Snowballing We used the guidelines of Wohlin [51] to perform snowball sampling to search the relevant literature that might have been missed by our search string.”
MethodInfluentialDávid, I., Latifaj, M., Pietron, J., Zhang, W., Ciccozzi, F., Malavolta, I., Raschke, A., Steghöfer, J.-P., & Hebig, R. (2022) · Software & Systems Modeling · doi:10.1007/s10270-022-01010-3 · Cited by 61
“Then, we also perform forward snowballing, i.e., selecting relevant studies among those citing one of the initially selected academic studies [89].”
BackgroundMethodInfluentialModel-driven engineering for mobile robotic systems: a systematic mapping study
Casalaro, G. L., Cattivera, G., Ciccozzi, F., Malavolta, I., Wortmann, A., & Pelliccione, P. (2021) · Software & Systems Modeling · doi:10.1007/s10270-021-00908-8 · Cited by 50
“Note that we perform a closed recursive backward and forward snowballing activity, as shown in [67].”
MethodThe Role of FAIR Data towards Sustainable Agricultural Performance: A Systematic Literature Review
Ali, B., & Dahlhaus, P. G. (2022) · Agriculture · doi:10.3390/agriculture12020309 · Cited by 48
“Following eligibility criteria, an additional search was conducted through Google Scholar [33], applying the ‘snowballing’ (or ‘backward and forward snowball’) technique [34].”
MethodSensing Technology Applications in the Mining Industry—A Systematic Review
Duarte, J., Rodrigues, F., & Branco, J. C. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19042334 · Cited by 43
“In the second research stage, all of the studies before 2017 were assessed, as proposed in the snowballing technique [26].”
BackgroundMethodScaling Climate Smart Agriculture in East Africa: Experiences and Lessons
Kirina, T. K., Groot, A., Shilomboleni, H., Ludwig, F., & Demissie, T. D. (2022) · Agronomy · doi:10.3390/agronomy12040820 · Cited by 59
Capabilities and Practices in DevOps: A Multivocal Literature Review
Amaro, R., Pereira, R., & da Silva, M. M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3166626 · Cited by 43
“In the next phase a snowballing [59] is conducted leading to extra 14 relevant publications found, which increased the total number of papers.”
BackgroundRoberts, B. R. T., MacLeod, C. M., & Fernandes, M. A. (2022) · Psychological Bulletin · doi:10.1037/bul0000360 · Cited by 64
“Our second method was “ backward snowballing ” (Wohlin, 2014): Articles found by the fi rst method were used to fi nd additional articles by mining their reference lists.”
MethodReinforcement Learning in Construction Engineering and Management: A Review
Asghari, V., Wang, Y., Biglari, A. J., Hsu, S., & Tang, P. (2022) · Journal of Construction Engineering and Management · doi:10.1061/(asce)co.1943-7862.0002386 · Cited by 58
State of the Art in AAC: A Systematic Review and Taxonomy
Curtis, H., Neate, T., & Gonzalez, C. V. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544810 · Cited by 60
40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study
Wyrich, M., Bogner, J., & Wagner, S. (2023) · ACM Computing Surveys · doi:10.1145/3626522 · Cited by 77
“The nine papers in the start set were published in IEEE TSE (5) Finally, a good start set “should not be too small” [64].”
InfluentialCorgo, J., Cruz, S. S., & Conceição, P. S. (2024) · AMBIO · doi:10.1007/s13280-024-02052-1 · Cited by 65
“Backward snowballing involved examining the references cited within those articles, while forward snowballing involved tracking forward citations of the initial articles (Wohlin 2014).”
Khan, A. A., Bello, A. O., Arqam, M., & Ullah, F. (2024) · Technologies · doi:10.3390/technologies12100185 · Cited by 73
Writing for synthesis of evidence in empirical software engineering
Wohlin, C. (2014) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2652524.2652559 · Cited by 46
“From the reference list, it is possible to conduct a backward snowballing [18] to identify more relevant papers.”
BackgroundMethodSelf-citationIdentification and prioritization of SLR search tool requirements: an SLR and a survey
Al-Zubidy, A., & Carver, J. C. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9626-5 · Cited by 76
“…a large amount of manual effort to identify relevant studies (Engström et al. 2008); – Researchers have difficulty defining keywords and terms when building search strings due to the lack of formality and consistent terminology in SE (Wohlin 2014b; Carver et al. 2013; Sjoberg et al. 2007); and –”
A systematic literature review of model-driven security engineering for cyber–physical systems
Geismann, J., & Bodden, E. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110697 · Cited by 61
“We applied forward as well as backward snowballing (Wohlin, 2014) until no additional papers were found.”
MethodModeling of IoT devices in Business Processes: A Systematic Mapping Study
Torres, V., Serral, E., Valderas, P., Pelechano, V., & Grefen, P. (2020) · IEEE Conference on Business Informatics · doi:10.1109/cbi49978.2020.00031 · Cited by 47
“In addition, to reduce the probability of missing relevant studies, we have complemented the initial search with backward and forward snowballing [19].”
BackgroundCaristia, S., De Vito, M., Sarro, A., Leone, A., Pecere, A., Zibetti, A., Filigheddu, N., Zeppegno, P., Prodam, F., Faggiano, F., et al. (2020) · Nutrients · doi:10.3390/nu12082290 · Cited by 42
“Potentially relevant literature and other eligible studies were further identified using a snowballing approach by screening the reference lists of all previously identified studies [38].”
BackgroundSharma, K., Torrado, J. C., Gómez, J., & Jaccheri, L. (2020) · Entertainment Computing · doi:10.1016/j.entcom.2020.100387 · Cited by 60
Nascimento, E., Nguyen‐Duc, A., Sundbø, I., & Conte, T. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03751 · Cited by 42
Dynamic capabilities for the creation of logistics flexibility – a conceptual framework
Sandberg, E. (2021) · The International Journal of Logistics Management · doi:10.1108/ijlm-07-2020-0266 · Cited by 55
Villalobos, N. V. F., Ott, J. J., Klett-Tammen, C. J., Bockey, A. J., Vanella, P., Krause, G., & Lange, B. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01732-3 · Cited by 46
“We further applied the snowball method [29] to available systematic reviews to identify additional evidence.”
MethodAn overview of machine learning techniques in constraint solving
Popescu, A., Erdeniz, S. P., Felfernig, A., Uta, M., Atas, M., Le, V.-M., Pilsl, K., Enzelsberger, M., & Tran, T. N. T. (2021) · Journal of Intelligent Information Systems · doi:10.1007/s10844-021-00666-5 · Cited by 62
“Following the snowballing technique (Wohlin, 2014), we further analyzed the reference sections of the identified papers with the goal to find further topic-related work.”
MethodIoT Serverless Computing at the Edge: A Systematic Mapping Review
Kjorveziroski, V., Filiposka, S., & Trajkovik, V. (2021) · Computers · doi:10.3390/computers10100130 · Cited by 49
“During this full-text analysis, 10 additional papers were identified by applying the snowballing technique [37] scouring the references of the read papers for additional relevant content.”
MethodPersonalising Explainable Recommendations: Literature and Conceptualisation
Naiseh, M., Jiang, N., Ma, J., & Ali, R. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-030-45691-7_49 · Cited by 42
“We followed the guidelines provided by Wohlin [53] for conducting systematic literature reviews using snowballing approach.”
MethodBarreñada, L., Ledger, A., Dhiman, P., Collins, G., Wynants, L., Verbakel, J. Y., Timmerman, D., Valentin, L., & Van Calster, B. (2024) · BMJ Medicine · doi:10.1136/bmjmed-2023-000817 · Cited by 42
Applicability of Physiological Monitoring Systems within Occupational Groups: A Systematic Review
Bustos, D., Guedes, J. C., Baptista, J. S., Vaz, M. A. P., Costa, J. T., & Fernandes, R. J. (2021) · Sensors · doi:10.3390/s21217249 · Cited by 52
Agapiou, A. (2022) · Buildings · doi:10.3390/buildings12030283 · Cited by 54
“The literature search was expanded through snowballing; reviewing the citations and reference list of each paper and further literature in hich the original paper has been cited [22].”
BackgroundMethodOn the relation between ‘resilience’ and ‘smartness’: A critical review
Cañavera-Herrera, J. S., Tang, J., Nochta, T., & Schooling, J. M. (2022) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2022.102970 · Cited by 61
Predictive Analytics for Early-Stage Construction Costs Estimation
Miranda, S. L. C., Castillo, E. D. R., González, V. A., & Adafin, J. (2022) · Buildings · doi:10.3390/buildings12071043 · Cited by 69
“A backward and forward snowballing process was performed on the 30 articles following the previous approach and following the suggestions provided by Wohlin [42].”
MethodMachine Learning for Software Engineering: A Tertiary Study
Kotti, Z., Galanopoulou, R., & Spinellis, D. D. (2023) · ACM Computing Surveys · doi:10.1145/3572905 · Cited by 67
Sentia, P. D., Shukor, S. A., Wahab, A. N. A., & Mukhtar, M. B. (2023) · Annals of Operations Research · doi:10.1007/s10479-023-05232-6 · Cited by 46
“This research proposes a methodology called a sequential thematic literature review method, which combines three stages namely the literature search process (Schutt, 2019); the key paper snowballing process (Wohlin, 2014); and the problem-cause solution order for determining themes.”
MethodInfluentialBig Data and Machine Learning in Defence
Weng, Y., & Wu, J. (2024) · International Journal of Computer Science and Information Technology · doi:10.5121/ijcsit.2024.16203 · Cited by 92
“Additionally, the reference lists of the included sources will be manually searched to identify potential additional relevant studies (backward reference searching) [16].”
Background
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.