Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 2,301–2,400 of 3,934
González-Jarrín, P., Fernández-Fernández, J., Tormo, J. V. G., & García, C. G. (2025) · Journal of Functional Morphology and Kinesiology · doi:10.3390/jfmk10040398 · Cited by 6
“Backward and forward snowballing techniques [34] were used to identify additional studies.”
Navigating challenges in subspecies management: a tale of two rabbits in iberia
Vaquerizas, P. H., Fa, J. E., Delibes‐Mateos, M., Castro, F., & Villafuerte, R. (2025) · European Journal of Wildlife Research · doi:10.1007/s10344-025-01978-5 · Cited by 5
“Given the potential for keyword-based searches to miss rel - evant studies due to inconsistent terminology across publications, we first employed a snowball sampling with no restrictions (Wohlin 2014).”
Ebrahim, K. M. P., Gomaa, S. M. M. H., Zayed, T. M., & Alfalah, G. (2025) · Water · doi:10.3390/w17223273 · Cited by 18
Delgado, L. M., Pompeu, B. F., Macedo, V. D. S., Martins, G. H. A., Pasqualotto, E., Araújo, M. R., Brunini, J. H., Nunes, V. A., Theis, C., & Aguiar, S. (2026) · Journal of Surgical Oncology · doi:10.1002/jso.70194 · Cited by 4
Ali, T., Mahmood, M. S., Ahmed, H. H., Qureshi, M. Z., Elhadi, K. M., Alaneme, G. U., Kırgız, M. S., & Ajwad, A. (2026) · International Journal of Concrete Structures and Materials · doi:10.1186/s40069-026-00907-3 · Cited by 12
Valorization of Food Processing By-Products: A Sustainable Approach for a Circular Bioeconomy
Gonal, B. N., Ravikumar, S. K., Pattanad, V. B., Patil, R. L., & Arakera, S. B. (2026) · Food Ethics · doi:10.1007/s41055-026-00208-1 · Cited by 4
“To complement database-driven keyword matching, which can sometimes fail due to inconsistent terminology in emerging fields, the “snowballing” methodology proposed by Claes Wohlin ( 2014 ) was adopted.”
Alves, L. F., Vasconcellos, F. J. S., & Nogueira, B. M. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10084-4 · Cited by 4
“Another complementary technique that has widely used in the identification of primary studies is snowballing, described by Wohlin (2014).”
BackgroundMethodInfluentialBleitz, M., Walters, K., & Latimore, J. A. (2024) · Lake and Reservoir Management · doi:10.1080/10402381.2023.2297231 · Cited by 4
“Unlike Wohlin (2014), we did not examine the complete list of referenced papers, but instead identified in-text citations that supported important relevant information, which we then compiled into an abbreviated list of referenced papers.”
BackgroundMethodA systematic review of refactoring opportunities by software antipattern detection
Kalhor, S., Keyvanpour, M. R., & Salajegheh, A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00443-y · Cited by 11
Carvalho, L. D. O., Biazotto, J. P., Feitosa, D., & Nakagawa, E. Y. (2024) · Congresso Ibero-Americano em Engenharia de Software (CIbSE) · doi:10.5753/cibse.2024.28456 · Cited by 6
“For this preliminary study that offers such an overview, we decided not to perform a snowballing-based review [Wohlin 2014].”
MethodTeacher absenteeism: a conceptual model developed from a systematic literature review
Carrizosa, L. M. G., & Witte, K. D. (2024) · Cambridge Journal of Education · doi:10.1080/0305764x.2024.2397366 · Cited by 11
“This technique refers to extending a systematic literature study, considering relevant articles as a reference point and then identifying additional papers to include (Wohlin, 2014).”
Willoughby, K., Atallah, S., Arbeau, K., Pearce, J., Ketelaars, T., & Pierre, J. S. (2024) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph21111441 · Cited by 6
A Study of Factors That Influence the Software Project Success
Monserrat, M., Mas, A., & Mesquida, A. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2735 · Cited by 4
Eaton, C., Campo, K. X., Longhurst, A., Mesmer, B. L., & Ferreira, S. (2024) · Systems Engineering · doi:10.1002/sys.21799 · Cited by 4
Gender Diversity in Software Development Teams: A Systematic Literature Review
Alarcón-Alarcón, J., Pérez-Arriaga, J. C., Sánchez-García, Á. J., & Hemández-González, L. A. (2024) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft63288.2024.00047 · Cited by 6
“2) Snowballing Process: We carried out a backward snowballing process following the proposal described in [8].”
Scott, M., DeLone, W. H., & McLean, E. R. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2452900 · Cited by 7
Akbar, M. A., Naseem, A., Zaman, U. K. U., & Petronijević, J. (2025) · International Journal of Management Science and Engineering Management · doi:10.1080/17509653.2025.2461001 · Cited by 4
“The fifth and last step for inclusion of finally selected articles came up with a total of 26 articles, including the one added through the forward snowball search technique in the light of guidance by Wohlin (2014).”
A comprehensive survey of manual and dynamic approaches for cybersecurity taxonomy generation
Spyros, A., Kougioumtzidou, A., Papoutsis, A., Darra, E., Kavallieros, D., Tziouvaras, A., Tsikrika, T., Vrochidis, S., & Kompatsiaris, I. Y. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02382-w · Cited by 11
“The snowballing methodology adopted in this study follows the guidelines of implementing snowballing in systematic literature studies, as described in relevant works in the literature [11].”
InfluentialThe qualitative factor in software testing: A systematic mapping study of qualitative methods
Ardic, B., Brandt, C., Khatami, A., Swillus, M., & Zaidman, A. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112447 · Cited by 13
Effect of olive oil consumption on diabetes risk: a dose-response meta-analysis
Du, Y., & Zhou, H. (2025) · Journal of Health, Population and Nutrition · doi:10.1186/s41043-025-00866-7 · Cited by 5
“To enhance the comprehensiveness of literature search, we conduced backward and forward snowballing searches to identify additional relevant articles [18].”
Understanding refactorings in Elixir functional language
Vegi, L. F. D. M., & Valente, M. T. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10652-y · Cited by 6
Dormohamadi, M., Kanafani, K., Birgisdóttir, H., Dragsted, A., & Hoxha, E. (2025) · Cleaner Environmental Systems · doi:10.1016/j.cesys.2025.100280 · Cited by 4
On the usage of artificial intelligence in leprosy care: A systematic literature review
de Andrade, H. G. V., Rocha, É. D. S., Monteiro, K. H. D. C., de Morais, C. M., dos Santos, D. C. M., Nascimento, D. C., Dourado, R. A. D. S., Lynn, T., & Endo, P. T. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012550 · Cited by 15
“…IEEE Xplorer, Pubmed, Science Direct, Scopus and Springer Link on the use of artificial intelligence to address key aspects in leprosy care, as defined by the World Health Organization [7] ( I1 ); and relevant studies cited by the works selected based on I1 (forward snowball search) [18] ( I2 ).”
Applications of AI in Space Domain
dos Santos, L. B. R., Basciani, F., & Pelliccione, P. (2025) · ACM Computing Surveys · doi:10.1145/3757317 · Cited by 5
“In the 4th stage, we performed a backward snowballing [57] on those 22 studies of our set.”
Domain Knowledge in Requirements Engineering: A Systematic Mapping Study
Araújo, M. C., Araújo, J., de Oliveira, R. F., Romão, L., & Kalinowski, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04200-2_29 · Cited by 7
Artificial intelligence for source code understanding tasks: A systematic mapping study
Fudholi, D. R., & Capiluppi, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107915 · Cited by 7
Khan, M., Akbar, M. A., & Kasurinen, J. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3727967.3756831 · Cited by 18
“Following the principles of evidence-based software engineering, we ensure a rigorous review process by systematically examining all primary studies related to the research questions in an unbiased and iterative way [35].”
Agarwal, B., Urlings, C. C., van Lankveld, G., & Klemke, R. (2026) · International Journal of Artificial Intelligence in Education · doi:10.1016/j.ijaied.2026.100004 · Cited by 11
Software Development Analytics in Practice: A Systematic Literature Review
Caldeira, J., Abreu, F. B. E., Cardoso, J., Simões, R. V., Oliveira, T., & dos Reis, J. P. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-022-09864-y · Cited by 10
Architecting Internet of Things Systems with Blockchain
Yánez-Pazmiño, W., Bahsoon, R., Zhang, Y., & Kazman, R. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3442412 · Cited by 10
“(6) Snowballing: We complemented our full-text reading with recursive backward and forward snowballing activities described by Wohlin [62] to complement the automatic search.”
MethodSuccess factors and challenges in digital forensics for law enforcement in Sweden.
Mori, M. D. C., Kävrestad, J., & Nohlberg, M. (2021) · Cited by 3
Park, J.-H., Shin, H., Kim, J., et al. (2021) · Korean Journal of Clinical Pharmacy · doi:10.24304/kjcp.2021.31.4.257 · Cited by 4
Using Database Schemas of Legacy Applications for Microservices Identification: A Mapping Study
Mparmpoutis, A., & Kakarontzas, G. (2022) · International Conference on Algorithms, Computing and Systems (ICACS) · doi:10.1145/3564982.3564995 · Cited by 4
“In this method, each research paper’s list of references is examined in the terms of the previously applied inclusion/exclusion criteria [24].”
MethodTasoudis, P., Vitkos, E. N., Haithcock, B. E., et al. (2023) · Surgical Endoscopy · doi:10.1007/s00464-023-09931-w · Cited by 4
A research landscape on software defect prediction
Taskeen, A., Khan, S. U. R., & Felix, E. A. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2549 · Cited by 10
Surveying the Academic Literature on the Use of OKR (Objective and Key Results)
Silva, R. V., & Santos, G. (2023) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3592813.3592934 · Cited by 11
“Moreover, the results obtained were complemented by performing backward and forward snowballing [27, 47] to provide further robustness to the search and increase the number of found works.”
BackgroundMethodde Souza, B. P., dos Santos, P. S. M., & Travassos, G. H. (2023) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3592813.3592927 · Cited by 4
“In addition, Snowballing (backward and forward) technique also contributed to the final selected papers [31].”
MethodSoK: Machine Learning for Continuous Integration
Arani, A. K., Zahedi, M., Le, T. H. M., & Babar, M. A. (2023) · doi:10.1109/aiops59134.2023.00006 · Cited by 7
“methods for reducing the risk of missing studies [14], we obtained a final list of 44 related studies.”
MethodRadan, S., Hassan, S. M., Wilson, A. S., & Basurra, S. S. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/978-1-6684-9072-3.ch001 · Cited by 5
Croucher, C., Powell, W. A., Stevens, B., Dicks, M., Powell, V., Wiltshire, T. J., & Spronck, P. H. M. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3313439 · Cited by 5
“Next, a backwards snowballing approach was adopted [61].”
BackgroundMethodCunha, L., Mora, M. M. R., Afzal, F., Cesar, G. M., Guimarães, C. R., Pontes, J. P. M., Alves, G. G., & Silveira, A. C. F. (2024) · European Journal of Pediatrics · doi:10.1007/s00431-024-05517-3 · Cited by 4
“The references from included studies and previous systematic reviews were also searched manually for any additional studies [32].”
Sentiment Analysis for Requirements Elicitation from App Reviews: A Systematic Mapping Study
Wan, H., An, Z., Wang, B., & Xiong, T. (2023) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00030 · Cited by 4
“We use the snowballing process proposed in [18] to search for missing papers, and iteratively searching the reference lists and cited papers in the literature until no new papers are found (step E in Fig.”
MethodDominant risk factors (DRFs) in construction-specific supply chains: a systematic review
Kadan, R., & Wium, J. A. (2024) · Frontiers in Engineering and Built Environment · doi:10.1108/febe-06-2023-0030 · Cited by 5
Reading abilities in preterm children: a systematic review and meta-analysis
Gutiérrez‐Ortega, M., Álvarez-Bardón, A., Vergara‐Moragues, E., Tubío, J., & González-Andrade, A. (2024) · Cognitive Processing · doi:10.1007/s10339-024-01192-2 · Cited by 9
“After the selection of the full articles, the guideline-based snowball approach was performed to find the articles that were not extracted by using the search strings (Wohlin 2014).”
Ali, O., Murray, P., Al‐Ahmad, A., & Tahat, L. H. (2024) · Emerging Science Journal · doi:10.28991/esj-2024-08-03-025 · Cited by 14
“A backward snowballing approach was employed to identify articles overlooked in the initial process by reviewing article reference lists [82].”
MethodA Meta-Model-Driven Approach to Support Digitization Evolution with Maturity Models
Bernabei, M., Costantino, F., & Tronci, M. (2024) · Journal of Industrial Integration and Management · doi:10.1142/s2424862224500143 · Cited by 4
Soares, K. D. N., Ribeiro, P. V. D. S., & Neto, M. G. D. S. (2024) · Revista Brasileira de Computação Aplicada · doi:10.5335/rbca.v16i2.15491 · Cited by 37
Palavani, L. B., Mitre, L. P., Almeida, M. V., et al. (2024) · Neurosurgical Review · doi:10.1007/s10143-024-02878-z · Cited by 4
“Additionally, 5 studies were included as a product of snowballing search [20].”
The Terminology of Automotive Product-Structuring Concepts: A Systematic Mapping Study
Zellmer, P., Holsten, L., Krüger, J., & Leich, T. (2024) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2024.3463179 · Cited by 4
Time, A., Pacheco, D. A. D. J., & Ionescu, S. C. (2024) · Information Development · doi:10.1177/02666669241287499 · Cited by 8
“Jalali and Wohlin (2012) argue that an adequate literature analysis must necessarily include various approaches to fi nding relevant papers, and it is a relatively common practice in literature review research (Blue et al., 2021; Wohlin, 2014).”
The Value of Applying Big Data Analytics in Health Supply Chain Management
Nuaimi, D. A., & Awofeso, N. (2024) · F1000Research · doi:10.12688/f1000research.156525.1 · Cited by 2
“In a snowball approach, both references in the paper (backward references) and papers referring to the paper (forward references) were screened (Wohlin, 2014).”
InfluentialMalabusini, S., & Lupi, D. (2024) · Insects · doi:10.3390/insects15110880 · Cited by 4
“In addition, a snowball referencing search [46–48] was carried out from articles collected from previous results.”
Cultural influence on RE activities: An extended analysis of state of the art
Muzammel, C. S., Spichkova, M., & Harland, J. (2024) · International Conference on Mobile Human-Computer Interaction (MobileHCI) · doi:10.1145/3640471.3680236 · Cited by 4
“We refined our SLR list by doing forward and backward snowballing iterations by the guidelines [38].”
Zoupas, I., Loufopoulos, G., Tasoudis, P., et al. (2024) · Asian Cardiovascular and Thoracic Annals · doi:10.1177/02184923241301108 · Cited by 4
“14 The Covidence reference and article manager software were used for the search and study collection.”
Aldin, A., Baumeister, A., Chakraverty, D., Monsef, I., Noyes, J., Kalbe, E., Woopen, C., & Skoetz, N. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013302.pub2 · Cited by 21
Makrakis, D., Msaouel, P., Karam, J. A., et al. (2024) · European Urology Focus · doi:10.1016/j.euf.2024.11.007 · Cited by 4
Vivanco‐Suarez, J., Woodiwiss, T. R., Fiock, K. L., et al. (2024) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2024.107243 · Cited by 4
Chaudhary, A., Javed, A., Colombo, W., & Manchego, F. A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-77524-6_1 · Cited by 8
Recommender Systems for Unified Modeling Language and Vice Versa—A Systematic Literature Review
Marand, E. A., Sheikhahmadi, A., Challenger, M., Moradi, P., & Khalilipour, A. (2025) · IEEE Access · doi:10.1109/access.2025.3535527 · Cited by 7
Wang, Y., Huang, Y., Li, J., Li, S., Zhang, H., Lyu, J., Zhong, C., Liu, X., Liu, B., Liu, Y., et al. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3538612 · Cited by 6
“In addition, snowballing [36] was introduced to mitigate the risk of missing important studies.”
Design practices in visualization driven data exploration for non-expert audiences
Tylosky, N., Knutas, A., & Wolff, A. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100731 · Cited by 7
Characterising harmful API uses and repair techniques: Insights from a systematic review
Ochoa, L., Hammad, M., Giray, G., Babur, Ö., & Bennin, K. E. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100732 · Cited by 6
Alqadi, M., Zaharieva, S., Commichau, A., Disse, M., Koellner, T., & Chiogna, G. (2025) · Sustainability · doi:10.3390/su17073121 · Cited by 8
The Evolving Role of Product Manager - A Systematic Review
Parikh, N. A. (2025) · Foundations of Management · doi:10.2478/fman-2025-0003 · Cited by 4
A systematic mapping study on automated negotiation for autonomous intelligent systems
Memon, M. A., Scoccia, G. L., & Autili, M. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00515-7 · Cited by 11
“It is worth mentioning that this step was not only focused on extracting all relevant studies but, as suggested in Wohlin (2014), we concentrated on extracting relevant studies to utilize as a basis for a subsequent snowballing procedure, discussed in the following.”
Hautte, L. V., Scheerens, C., Ruyssen, I., & Toma, S. (2025) · Globalization and Health · doi:10.1186/s12992-025-01116-5 · Cited by 9
“These strategies were adopted to ensure the retrieval of all relevant articles and to maximize the comprehensiveness of the scoping review [22].”
Adamu, U. G., Mojela, K., Adamu, E.-A. U., Ojji, D. B., & Tsabedze, N. (2025) · High Blood Pressure & Cardiovascular Prevention · doi:10.1007/s40292-025-00720-1 · Cited by 5
“The results of the literature search were imported into Rayyan software, and the duplicates were removed [ 12].”
Massari, A., Peroni, S., Tomasi, F., & Heibi, I. (2026) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqaf076 · Cited by 6
“In order to conduct the review, we adopted a citation-based strategy (Figure 1), also known as “snowballing” (Wohlin, 2014), which consists of exploding the bibliography from a seed paper (Lecy & Beatty, 2012), which was (Sikos & Philp, 2020).”
Investigating factors that influence developers' experience in mobile software ecosystems
Fontão, A., Dias‐Neto, A. C., & Viana, D. (2017) · doi:10.5555/3103196.3103209 · Cited by 4
Technical Debt Management in Brazilian Software Organizations: A Need, an Expectation, or a Fact?
da Silva, V. M. M., Jeronimo, H., & Travassos, G. H. (2018) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3275245.3275267 · Cited by 6
“That is the case of DebtFlag tool, presented first in [29] with no evidence and then evaluated in [30].”
MethodFreire, A., Meireles, A., Guimarães, G., et al. (2018) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2018-185 · Cited by 4
“Therefore, to mitigate this problem, we executed the snowballing technique, as described in [15], until no more relevant papers were found.”
MethodInfluentialSoftware Product Line to overcome the variability issue in E-Learning
Sebbaq, H., Retbi, A., Idrissi, M. K., & Bennani, S. (2018) · International Conference on Intelligent Systems: Theories and Applications · doi:10.1145/3289402.3289504 · Cited by 4
“At the end of this 1st iteration, we apply the procedure of "Snowballing" [2] on our selection of sources.”
BackgroundMethodInfluentialBusiness Process Management in Digital and Software Ecosystems: A Systematic Mapping Study
Afonso, A. T. Q., Chueri, L., & dos Santos, R. P. (2020) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c50368.2020.00046 · Cited by 7
Siqueira, B. R., Ferrari, F. C., de Souza, K. E., Santibanez, D. S. M., & de Camargo, V. V. (2020) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw50294.2020.00054 · Cited by 6
“Search Strategy: It comprises: (i) performing automatic search; and (ii) applying backward and forward snowballing [40].”
MethodOracles of Bad Smells: a Systematic Literature Review
Trindade, R. P. F., Bigonha, M. A. S., & Ferreira, K. A. M. (2020) · Brazilian Symposium on Software Engineering · doi:10.1145/3422392.3422415 · Cited by 6
“Then, we conducted the procedure known as backward snowballing [57] with the 46 articles.”
MethodCloud management and monitoring - a systematic mapping study
Odun-Ayo, I. A., Williams, T.-A., & Yahaya, J. B. (2021) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v21.i3.pp1648-1662 · Cited by 6
“In addition, the backward snowballing process is adopted to refine the search [26].”
MethodModelling on mobile devices: A systematic mapping study
Brunschwig, L., Guerra, E., & de Lara, J. (2021) · Software & Systems Modeling · doi:10.1007/s10270-021-00897-8 · Cited by 5
Wong, Y. Y. B., Joyce, S. C., & Blessing, L. T. M. (2021) · Proceedings of the Design Society · doi:10.1017/pds.2021.21 · Cited by 4
“The next step of the literature review was a backward snowballing approach (Wohlin 2014).”
BackgroundLiterature Grounded Theory (LGT)
Ermel, A. P. C., Lacerda, D. P., Morandi, M. I. W. M., & Gauss, L. (2021) · Springer eBooks · doi:10.1007/978-3-030-75722-9_6 · Cited by 8
Ajam, G., Rodríguez, C., & Benatallah, B. (2021) · CLEI electronic journal · doi:10.19153/cleiej.24.2.5 · Cited by 10
“Backward snowballing [51] was also applied in order to find studies related to API documentation that were missed through the search strategy above.”
MethodCorrêa, C. M., & Silveira, M. S. (2021) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3472301.3484332 · Cited by 4
“Following the rules stated by Wohlin [49], to avoid bias in favor of any specifc publisher, we used Google Scholar to defne the initial set of studies.”
MethodA brief review of dashboard visualizations employed to support management or business decisions
Frazao, D. A. G., da Costa, T. S. A., de Araujo, T. D. O., Meiguins, B. S., & dos Santos, C. G. R. (2021) · International Conference on Information Visualisation (IV) · doi:10.1109/iv53921.2021.00025 · Cited by 9
“This review used the forward Snowballing [9] [18] [19] method to find papers that used dashboard visualizations in the context of management or business.”
MethodBlockchain Driven Access Control Mechanisms, Models and Frameworks: A Systematic Literature Review
Dar, A. B., Lone, A. H., Naaz, R., Baba, A. I., & Wu, F. (2022) · Journal of Information Security and Cybercrimes Research · doi:10.26735/tyif2196 · Cited by 7
“These criteria helped in attaining the results, which were then put through Wohlin’s snowballing process [10].”
BackgroundTowards comprise the Design System Applicability: A Multivocal Literature Mapping
Vendramini, S., Belusso, L., Souza, F. C. M., & Souza, A. C. C. (2021) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3493244.3493260 · Cited by 9
“To ensure all the relevant source’s inclusion, we conducted backward snowballing [12], as recommended by systematic review guidelines, on the set of papers already in the pool.”
MethodThe Changing Role of the Transport Facility Internal Audit in the Post-COVID Period
Krivtsov, A. I., & Tarasova, T. M. (2022) · Springer eBooks · doi:10.1007/978-3-030-83566-8_13 · Cited by 4
A systematic literature review and taxonomy of modern code review
Davila, N., & Ingrid, N. (2021) · doi:10.26226/morressier.613b5419842293c031b5b639 · Cited by 7
The Adoption of Artificial Intelligence in the South African E-Commerce Space: A Systematic Review
Malapane, T. A., & Ndlovu, N. K. (2022) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds55548.2022.9799403 · Cited by 6
Um Estudo sobre Observatórios através de um Mapeamento Sistemático da Literatura
Vieira, J. K. M., Barbosa, J. L. P., de Farias, I. H., et al. (2022) · Journal of Information Systems and Technology Management · doi:10.4301/s1807-1775202219003 · Cited by 4
Software security management in critical infrastructures: a systematic literature review
EKŞİ, G. E., Tekinerdoğan, B., & Catal, C. (2022) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.3841 · Cited by 5
“At the end of this process, we conduct snowballing technique [22] to gain more relevant studies in the literature.”
MethodKulmer, F., Wolf, M., & Ramsauer, C. (2022) · Procedia CIRP · doi:10.1016/j.procir.2022.05.109 · Cited by 4
Revealing Agile Mindset Using LEGO SERIOUS PLAY: Experience from an Online Agile Training Project
Fronza, I., & Wang, X. (2022) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2022-117 · Cited by 5
Sin, B. S. Y., & Cheng, J. L. A. (2022) · Journal of Cognitive Sciences and Human Development · doi:10.33736/jcshd.4745.2022 · Cited by 8
“This comprehensive literature review uses the hybrid search strategy, which utilises both database search and snowballing methods to extract relevant articles (Wohlin, 2014; Wohlin et al., 2022).”
MethodFilho, J. C. D. S., Sampaio, A. L., & Feitosa, E. L. (2022) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3554364.3559136 · Cited by 4
“Based on the guidelines of Wohlin [62], we present the methodological procedures in the following subsections.”
MethodInfluentialMethods to Recommend Corrective Measures for Agricultural Soils: A Systematic Literature Study
Moreira, W. K. O., de Souza, E. G., Beneduzzi, H. M., Nóbrega, L. H. P., Bazzi, C. L., Maggi, M. F., Rodrigues, M., & Gavioli, A. (2022) · Communications in Soil Science and Plant Analysis · doi:10.1080/00103624.2022.2137194 · Cited by 5
“In this context, this study aimed to conduct a systematic literature study (SLS; Wohlin 2014) to synthesize evidence and understand the status of methods to recommend corrective measures for agricultural soils.”
BackgroundMethodThorn, J. R., Strandberg, P. E., Sundmark, D., & Afzal, W. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1131 · Cited by 6
“The literature study was based on guidelines on literature studies and snowballing (Kitchenham & Charters, 2007; Wohlin, 2014).”
MethodCategorizing Research on Identity in Undergraduate Computing Education
Kapoor, A., & Gardner‐McCune, C. (2022) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3564721.3565948 · Cited by 7
Mittal, M., & Manocha, S. (2022) · Journal of Information and Optimization Sciences · doi:10.1080/02522667.2022.2134367 · Cited by 8
“The researchers carried out a comprehensive search of web-based databases like ProQuest, Google Scholar, and Research Gate to identify relevant academic literature using the Snowball technique described by (Wohlin, 2014).”
MethodPaving the way to a software-supported requirements prioritization in distributed scrum projects
Model, K., Mombrey, C., & Herzwurm, G. (2022) · doi:10.1145/3524614.3528633 · Cited by 6
“To address these RQs while establishing a scientific as well as a practical research lens [24], and therefore to guarantee triangulation of data [9], we conducted a literature review using snowballing [36] combined with semi-structured expert interviews [21].”
MethodIncluding business strategy in model-driven methods: an experiment
Noël, R., Panach, J. I., & Pastor, Ó. (2023) · Requirements Engineering · doi:10.1007/s00766-023-00400-3 · Cited by 4
“We performed a backward snowballing [32] looking for relevant referenced studies, adding five more articles.”
MethodWEBAPIK: a body of structured knowledge on designing web APIs
Sadi, M. H., & Yu, E. S. K. (2023) · Requirements Engineering · doi:10.1007/s00766-023-00401-2 · Cited by 4
“For scholarly discussions, forward snow-balling process is also used [24] to examine the resources referenced in the retrieved resources.”
Method
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.