Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 2,701–2,800 of 3,934
NFR Evaluation in IoT Applications: Methods, Strategies and Open Challenges
Paiva, J. O. V., Andrade, R. M. C., & Carvalho, R. M. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-08965-7_15 · Cited by 2
Huttunen, H.-L., Seppänen, P., & Halonen, R. (2022) · Finnish Journal of eHealth and eWelfare · doi:10.23996/fjhw.111575 · Cited by 3
“To support the empirical part of our study, to understand the relevance of “Technology Acceptance Model in Healthcare”, and to explore the latest studies on wearable sensors, self-tracking and self-care, a literature search was conducted following guidelines suggested by Petersen et al. [36] and Wohlin [37], resulting…”
BackgroundPIS: Interoperability and Decision-Making Process—A Review
Fernandes, J. C., Oliveira, L., Neto, V. V. G., et al. (2022) · Springer eBooks · doi:10.1007/978-3-031-18176-4_7 · Cited by 2
Bots and their Uses in Software Development: A Systematic Mapping Study
Moguel-Sánchez, R., Palacios, C. S. M., Ocharán-Hernández, J. O., Limón, X., & Sánchez-García, Á. J. (2022) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft55708.2022.00027 · Cited by 8
“To expand on our results, future work includes an ongoing search for additional studies about bots in software development through Snowballing search process [24].”
BackgroundAn exploratory study on the critical features of construction project planning software
Okudan, O. (2023) · Sigma Journal of Engineering and Natural Sciences – Sigma Mühendislik ve Fen Bilimleri Dergisi · doi:10.14744/sigma.2023.00046 · Cited by 2
On knowing what is inside the walls – Challenges to Asset Management from a product perspective
Mohn, K., & Lohne, J. (2023) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/1176/1/012019 · Cited by 2
“Identified articles were utilized in the search using snowballing techniques (backwards and forwards), according to [27].”
MethodTowards a Framework for Self-adaptive Systems Conceptual Modeling
da Silva, J. P. S., Pimenta, M. S., Ecar, M., Giordano, D. M., Chagas, J. M., Otokovieski, M. B., Dias, B. D. A., & Forrati, S. M. (2023) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3592813.3592921 · Cited by 2
“We defined a set of related works by applying the snowballing technique (see [36]).”
MethodShi, J., Sitthiworachart, J., & Ratanaolarn, T. (2023) · World Journal of English Language · doi:10.5430/wjel.v13n7p154 · Cited by 3
Crowdfunding for Science and Teaching in Higher Education: Status Quo and Research Agenda
Wenzlaff, K. (2023) · Contributions to finance and accounting · doi:10.1007/978-3-031-30069-1_2 · Cited by 2
Business Process Reengineering in Agile Manufacturing – A Mixed Method Research
Lahlou, K., Medini, K., Wuest, T., et al. (2023) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-43662-8_24 · Cited by 2
A Gamified Approach to Engage Sustainable User Behavior in Product Service System Solutions
Villamil, C. (2023) · European Journal of Sustainable Development · doi:10.14207/ejsd.2023.v12n3p152 · Cited by 3
“Furthermore, a snowballing process helped to add key references (Wohlin, 2014), by selecting references that have been highly used in the research area.”
BackgroundInternet of Things - New Insights [Working Title]
Habib, M. K. (2023) · IntechOpen eBooks · doi:10.5772/intechopen.111033 · Cited by 3
de la Cruz, E. E., Prieto, J. M. M., & Maroto, A. (2023) · OBM Neurobiology · doi:10.21926/obm.neurobiol.2304191 · Cited by 2
NoSQL Database Modeling and Management: A Systematic Literature Review
Vera, R. A. A., Jácome, A. N., Mendoza, J. D., et al. (2023) · Revista Facultad de Ingeniería · doi:10.19053/01211129.v32.n65.2023.16519 · Cited by 2
“To carry out this SLR, the guide proposed by Kitchenham in [15] was used, and to increase the number of selected primary studies (SPS) that could be relevant, the authors decided to use the forward snowball method [16] in a complementary way to identify new articles in the first set of SPS.”
MethodBeyersdorffer, P., Münch, J., & Kuhrmann, M. (2023) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice/itmc58018.2023.10332295 · Cited by 2
“After three iterations, the snowballing procedure saturated and yielded only papers already included in our paper set 1 so that the termination condition was reached [21].”
MethodShopping Companions and Their Diverse Impacts: A Systematic Annotated Bibliography
Scholz, T. B., Pagel, S., & Henseler, J. (2023) · SAGE Open · doi:10.1177/21582440231221905 · Cited by 8
“Based on these 64 records, 28 more records that had not been returned by database searches could be identified employing the snowball technique (Wohlin, 2014).”
Nabi, M., Ilyas, M., & Ahmad, J. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2651 · Cited by 3
Koudounas, G., Giannopoulos, S., Houser, A., et al. (2024) · The Journal of Vascular Access · doi:10.1177/11297298241226993 · Cited by 2
Resilient Materials and Technologies for a Single-Family Home: A Review
Langar, S., & Kaduskar, S. (2024) · ASCE Construction Research Congress (CRC) · doi:10.1061/9780784485279.021 · Cited by 2
Gaps in strategic problem‐solving methods: A systematic literature review
Manso, D. F., Parnell, G. S., POHL, E. A., & Belderrain, M. C. N. (2024) · Journal of Multi-Criteria Decision Analysis · doi:10.1002/mcda.1828 · Cited by 4
Fritola, R. G., & Santander, V. F. A. (2022) · Workshop em Engenharia de Requisitos (WER) · doi:10.29327/1298262.25-28 · Cited by 2
“Após a finalização, será utilizada a técnica de Snowballing [19] para identificar artigos adicionais com base nas citações e por fim, uma classificação qualitativa dos artigos.”
A Review of Marine Debris Research in Vietnam
Nguyen, T. H. T. (2024) · Journal of Coastal Research · doi:10.2112/jcoastres-d-23-00085.1 · Cited by 2
“In this study, the literature search was conducted using the snowball method proposed by Wohlin (2014) with two following steps.”
MethodZabudkina, A., Lisein, O., & Pichault, F. (2024) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-52120-1_7 · Cited by 5
da Silva, F. N. R., de Souza, B. P., & Travassos, G. H. (2024) · Congresso Ibero-Americano em Engenharia de Software (CIbSE) · doi:10.5753/cibse.2024.28447 · Cited by 3
“An initial PICO-inspired search string (Table 1) was performed on Scopus (www.scopus.com) on December 22, 2022, returning 130 articles, with six selected to form the starting points for Snowballing (one level backward and forward) [Wöhlin 2014].”
MethodPaving a Path for a Combined Family of Feature Toggle and Configuration Option Research
Mahdavi-Hezaveh, R., Fatima, S., & Williams, L. A. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672555 · Cited by 4
“One example is TriPlayer [82].”
Nudging Interventions on Stair Use: A Scoping Review
Tzikas, A., Koulierakis, G., Athanasakis, K., & Merakou, K. (2024) · Journal of Prevention · doi:10.1007/s10935-024-00790-2 · Cited by 5
“Following a snowball method, we �rst used backward search and then we conducted a forward search (citation search) (Wohlin 2014).”
Functional Programming Oriented Software Design: A Systematic Literature Review
Camarillo-Villa, J. D., Limón, X., Cortés-Verdín, K., & Sánchez-García, Á. J. (2023) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft58849.2023.00015 · Cited by 2
“Wohlin [32] points out that, for the Snowballing process, a set of key studies (following their evaluation against the inclusion and exclusion criteria) should be employed to identify new articles that will be incorporated into the research.”
BackgroundTowards Effective People Management in Software Projects: A Comprehensive Catalog of Good Practices
Burkard, M., & Fontoura, L. M. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819402430001x · Cited by 2
Bridging Two Worlds: Framework for Secure Implementation of Blockchain Oracles
Iqbal, M., Chiarelli, A., & Matulevičius, R. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c62648.2024.00008 · Cited by 2
“We also utilize a snow-balling technique that helps us discover additional relevant literature studies by examining the initial set of collected studies [12].”
MethodPortela, L. J. P., Lauer-Leite, I. D., & de Novaıs, J. S. (2024) · psicología desde el caribe · doi:10.14482/psdc.40.3.720.589 · Cited by 3
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2024) · Springer eBooks · doi:10.1007/978-3-662-69306-3_4 · Cited by 4
Self-citationMotif discovery in hospital ward vital signs observation networks
Ironside-Smith, R., Noë, B., Allen, S. M., Costello, S., & Turner, L. D. (2024) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-024-00490-1 · Cited by 2
“Subsequently, we used the ‘snowball’ search method (Wohlin 2014) to identify additional works focused on ward-level outcomes in response to the procedure of undertaking vital sign observations (e.g., frequency and compliance to hospital policies).”
MethodA Review of Techniques for Reducing Shortcomings in Classical Information Visualization Charts
Lobato, N. F., de Araujo, T. D. O., Meiguins, B. S., et al. (2024) · International Conference Information Visualisation (IV) · doi:10.1109/iv64223.2024.00013 · Cited by 2
Quatrini, S., Hunter, E., Tindale, S. J., Newell‐Price, P., Frewer, L. J., & Lieberherr, E. (2024) · Agronomy · doi:10.3390/agronomy14112599 · Cited by 2
Ogilvie’s Syndrome Following Herpes Zoster Infection: A Comprehensive Review of the Literature
Loufopoulos, I., Waheed, U., Anestiadou, Ε., et al. (2024) · Cureus · doi:10.7759/cureus.74191 · Cited by 2
Hudson, J.-A. B. J., Shabbir, W., Hayawi, L. M., Chan, M. L. M., Barrowman, N. J., Sikora, L., & Ferretti, E. (2024) · Frontiers in Pediatrics · doi:10.3389/fped.2024.1450121 · Cited by 3
Syed, Z. A., Okegbola, O., & Akiotu, C. A. (2024) · Advances in electronic commerce (AEC) book series/Advances in electronic commerce series · doi:10.4018/979-8-3693-5966-2.ch010 · Cited by 9
The anatomical basis for surgical intervention in myositis ossificans—A systematic review
Stammer, A. T., Ashwood, N., Amara, V. V., et al. (2024) · Clinical Anatomy · doi:10.1002/ca.24238 · Cited by 2
Ferramentas para Snowballing: Uma Revisão Sistemática e Comparativa da Literatura
Pinheiro, C., Comis, D., & Rodrigues, E. (2024) · Escola Regional de Engenharia de Software (ERES) · doi:10.5753/eres.2024.4158 · Cited by 4
“A abordagem para examinar os artigos ´e similar `a utilizada para os artigos identificados por meio do snowballing reverso Wohlin 2014.”
InfluentialA structural taxonomy for lifted software product line analyses
Murphy, L., Saifi, M., Sandro, A. D., & Chećhik, M. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112280 · Cited by 3
Artificial Intelligence Methods in Software Refactoring: A Systematic Literature Review
Motogna, S., Berciu, L.-M., & Moldovan, V. (2024) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa64295.2024.00055 · Cited by 5
“In order to address threats to validity of our approach and to complement the automatic results with findings from different studies, the next step was to perform snowballing, as described in [17].”
Surveying the Academic Literature on the Use of OKR (Objectives and Key Results) – An Update
Silva, R. V., & Santos, G. (2024) · iSys - Brazilian Journal of Information Systems · doi:10.5753/isys.2024.3885 · Cited by 8
“Backward snowballing uses a study’s references to identify new sources that might be included, and forward snowballing relates to identifying sources that mention previously selected studies [Wohlin 2014].”
How Airport Service Quality and Accessibility are Mutually Dependent
Santos, I. D., Camarotto, J. A., & Tonin, L. A. (2025) · AIAA SciTech Forum · doi:10.2514/6.2025-2448 · Cited by 2
Delgado, L. M., Pompeu, B. F., Pompeu, B. F., et al. (2025) · Journal of Laparoendoscopic & Advanced Surgical Techniques · doi:10.1089/lap.2024.0382 · Cited by 2
A Meta-Analysis of the Efficacy of Conversational AI Use in EFL Learning Contexts
Gu, W. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-82915-4_14 · Cited by 3
Bispo, E. L., dos Santos, S. C., & Matos, M. V. A. B. D. (2025) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi_estendido.2025.246768 · Cited by 2
Wyke, S., & Lindhard, S. M. (2025) · Journal of Information Technology in Construction · doi:10.36680/j.itcon.2025.026 · Cited by 2
“A systematic literature review was conducted using the snowballing methodology (Wohlin, 2014), to attain an thorough understanding of the investigated topic.”
Recent progress on hydrogen-fueled port fuel injection spark ignition engine: A systematic review
Abubakar, S., Said, M. F. M., Abas, M. A., Khalid, A., Roslan, M. F., Ismail, N. A., Umaru, S., Narayan, S., & Kaisan, M. U. (2025) · Thermal Science · doi:10.2298/tsci240930081a · Cited by 4
“…articles were obtained through backward snowballing, where reference lists from some articles were used to identify some additional articles [45] - since the specific retrieval settings may result in the inadvertent exclusion of some pertinent publications -and were added which tallied the…”
Jensen, H. E., Ilyas, C. M. A., Tashk, A., et al. (2025) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3715669.3725897 · Cited by 2
“This review adhered to common practices in software engineering literature reviews, drawing on the methodologies of systematic mapping studies and snowballing as described by Petersen et al. [2015]; Wohlin [2014].”
Zero-carbon building materials in New Zealand context: a systematic literature review
Araghi, A. H., Rasheed, E. O., Vishnupriya, V., & Seadon, J. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-06564-7 · Cited by 5
“Besides, to ensure comprehensive coverage of relevant studies, this research employs the backward and forward snowballing technique [59].”
Possaghi, I., & Papavlasopoulou, S. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2513569 · Cited by 5
“Then, we applied the ‘snowballing’ method to reference lists and citations to find additional studies (Wohlin 2014).”
Assessing the Effectiveness of ChatGPT in Secure Code Development: A Systematic Literature Review
Bouzid, R., & Khoury, R. (2025) · ACM Computing Surveys · doi:10.1145/3744553 · Cited by 6
“Finally, we applied forward and backward snowballing [87, 88] to obtain eight additional sources.”
Cybersecurity Readiness Model Based on Human Factors
Abiodun, Y. T., Mahmood, S., Niazi, M., Alshayeb, M., & AlGhamdi, A. A. (2025) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-025-10349-w · Cited by 3
A Review of Quantum Machine Learning Applications
Marcozzi, M., Filatovas, E., & Paulavičius, R. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-88486-3_7 · Cited by 2
Lekscha, J., & Mirbabaie, M. (2025) · i-com · doi:10.1515/icom-2025-0005 · Cited by 6
Requirements Engineering for Machine Learning-Based AI Systems: A Tertiary Study
Martins, M. C., Campos, L. O., Soares, J. L. R., Kudo, T. N., & Bulcão-Neto, R. F. (2025) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2025.4892 · Cited by 7
“In addition, as an attempt not to leave out relevant secondary studies, we also performed backward and forward snowballing Wohlin (2014) over the studies’ citations and references resulting from papers’ full reading.”
The complex ecosystem of hyperprolific authors
Almeida, J. M., Antelmi, A., Gonçalves, M. A., & Pellegrino, M. A. (2026) · Scientometrics · doi:10.1007/s11192-026-05563-8 · Cited by 2
A Scoping Review and Guidelines on Privacy Policy's Visualization from an HCI Perspective
Zhang, S., He, E., Tao, S., Yang, Y., Ma, Y., Wang, A., Yi, X., & Li, H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790320 · Cited by 3
“To complement the search and ensure no omission of other papers, we performed forward and backward snowballing [198, 205], as well as consulting other researchers within the institution.”
A framework for analysing differences between public-sector and other-sector projects
Gasik, S. (2018) · Zarządzanie Publiczne · doi:10.15678/zp.2018.45.3.05 · Cited by 3
“Papers on intersectorial differences, both at the organisational as well as the project level, have been collected using the snowball approach (Wohlin, 2014).”
MethodDuarte, J., Vaz, M. A. P., Costa, J. T., et al. (2019) · International Journal of Occupational and Environmental Safety · doi:10.24840/2184-0954_003.002_0003 · Cited by 2
“If found, they will be used in new search combinations with the keywords previously used (Wohlin, 2014).”
MethodBridging the gap between software architecture and business model development: A literature study
Hyrynsalmi, S., Rauti, S., & Kaila, E. (2019) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.23919/mipro.2019.8756974 · Cited by 2
“From all articles, further relevant primary studies are selected with the backward snowballing technique [28].”
BackgroundRecent Trends in Software Quality Interrelationships: A Systematic Mapping Study
Shoga, M., Chen, C., & BOEHM, B. W. (2020) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c51114.2020.00052 · Cited by 2
“For each paper in the start set, we conducted a backward snowballing search on its references and a forward snow-balling search on its citations, following the guidelines in [20].”
MethodSoftware system deployment process: A systematic mapping study.
Panizzi, M. D., Genero, M., & Bertone, R. A. (2020) · Conferencia Iberoamericana de Software Engineering · Cited by 5
Measurement-based Experiments on the Mobile Web: A Systematic Mapping Study
de Munk, O., & Malavolta, I. (2021) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3463274.3463318 · Cited by 4
“The main goal of this phase is to complement the previously described automatic search with a snowballing activity [34].”
BackgroundMethodCase Study Protocol for the Analysis of Sustainable Business Models
Sánchez-Planelles, J., & Segarra‐Oña, M. (2021) · Springer proceedings in business and economics · doi:10.1007/978-3-030-72469-6_10 · Cited by 2
Exploring research hypotheses in green computing
Hamizi, I., Kholmatova, Z., & Succi, G. (2021) · Journal of Smart Environments and Green Computing · doi:10.20517/jsegc.2021.05 · Cited by 2
Data Collection in an IoT Off-Grid Environment Systematic Mapping of Literature
Goulart, A., Pinto, A. S. R., Boava, A., et al. (2022) · Sensors · doi:10.3390/s22145374 · Cited by 2
“During the analysis of these 16 works, the backward and forward snowballing process was performed for each of the [4] works.”
MethodİŞİ, N., & İŞİSAĞ, K. U. (2022) · Çeviribilim ve Uygulamaları Dergisi · doi:10.37599/ceviri.1114140 · Cited by 2
Mapping of Brazilian social housing studies: state of the art
Muianga, E. A. D., & Kowaltowski, D. C. C. K. (2022) · Encontro Nacional de Tecnologia do Ambiente Construído · doi:10.46421/entac.v19i1.2002 · Cited by 2
Inculcating the Peace Innovation Approach in Shaping the Future of Peace and Security in Africa
Maseno, M. K. (2023) · Springer eBooks · doi:10.1007/978-3-031-39043-2_2 · Cited by 2
Change Impact in Product Lines: A Systematic Mapping Study
Brink, C., Heisig, P., & Wackermann, F. (2016) · Communications in computer and information science · doi:10.1007/978-3-319-46254-7_55 · Cited by 3
On the Future of Solution Composition in Software Ecosystems
Yang, Z., Jansen, S., Gao, X., et al. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-61920-0_1 · Cited by 2
“After an extensive literature study, a couple of scientific frameworks were selected by conducting snowballing procedure [25].”
Design-Build and Design-Bid-Build in Construction- a Comparative Review
Plusquellec, T., Lehoux, N., & Cimon, Y. (2017) · Annual Conference of the International Group for Lean Construction · doi:10.24928/2017/0057 · Cited by 2
“Once those articles were found, the authors focused on their lists of references to identify other interesting articles related to the subject (i.e., backward snowballing, see Wohlin (2014)).”
BackgroundThe management of crowdsourcing software projects: A systematic mapping
de Deus, W. S., Fabri, J. A., & L’Erário, A. (2017) · Iberian Conference on Information Systems and Technologies (CISTI) · doi:10.23919/cisti.2017.7975711 · Cited by 2
“Já para a técnica do snowballing nós adotamos a abordagem de Wohlin [23].”
BackgroundProcess Improvement Archaeology: What Led Us Here, and What’s Next?
Unterkalmsteiner, M., & Gorschek, T. (2018) · IEEE Software · doi:10.1109/ms.2018.227110005 · Cited by 2
Explanations in Artificial Intelligence Decision Making: A User Acceptance Perspective
Vinson, N. G., Molyneaux, H., & Martin, J. (2019) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-5225-9069-9.ch006 · Cited by 2
“To illustrate how the principles relate to explanations, the principles are coupled with real world cases identified through backward snowballing (Wohlin, 2014) of the articles and reports collected and through Google searches.”
BackgroundMethodSociology in UK nurse education curricula: A review of the literature from 1919 to 2019
Molesworth, M., & LEWITT, M. S. (2019) · Social Theory & Health · doi:10.1057/s41285-019-00104-1 · Cited by 2
Planning the BIM Process in AEC Projects
Styrvold, M., Knotten, V., & Lædre, O. (2019) · Annual Conference of the International Group for Lean Construction · doi:10.24928/2019/0208 · Cited by 2
Technical Aspects of Software Development in Startups: A Systematic Mapping
Cavalcante, B. H., Leal, G. C. L., Balancieri, R., & Junior, I. D. F. (2018) · Latin American Computer Conference (CLEI) · doi:10.1109/clei.2018.00021 · Cited by 5
“A técnica consiste em verificar a lista dos artigos referenciados nos estudos aceitos com intuito de identificar novos artigos para inclusão [15].”
BackgroundPartnership Models for Software Ecosystems: A Systematic Mapping Study
Belo, Í., & Alves, C. F. (2019) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2019.00065 · Cited by 3
Ključni izzivi prilagajanja podnebnim spremembam v gradbenem sektorju
Kristl, Ž., Senior, C., & Salaj, A. T. (2020) · Urbani izziv · doi:10.5379/urbani-izziv-2020-31-01-004 · Cited by 3
Terzopoulos, G., Satratzemi, M., & Tsompanoudi, D. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-49932-7_50 · Cited by 2
da Cunha, J. A. O. G., Aguiar, Y. P. C., Pontes, J., et al. (2020) · doi:10.5753/washes.2020.11196 · Cited by 2
Vargas-Gomez, R., Pérez-Arriaga, J. C., Ocharán-Hernández, J. O., & Sánchez-García, Á. J. (2020) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft50191.2020.00032 · Cited by 3
“The process of snowballing was performed following the guidelines of Wohlin [9].”
BackgroundMethodHauck, J. C. R., & Leal, S. (2020) · Brazilian Symposium on Software Engineering · doi:10.1145/3422392.3422410 · Cited by 2
Factors Affecting Government Employees’ Acceptance of EDMS: A Systematic Review
Lojonon, B. G., & Alfred, R. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-33-4069-5_28 · Cited by 3
Musculoskeletal Ergonomic Implications in Smartphone Users: A Systematic Review
Vitorino, D. F., Correia, W. F. M., & Marçal, M. A. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-74611-7_117 · Cited by 2
“Thus, a systematic review of the literature was performed through the Scopus database, using the Snowballing approach [1] to form the most robust set of documents on the subject addressed.”
MethodSuárez-Álvarez, R., & Jiménez, A. G. (2021) · Doxa Comunicación Revista interdisciplinar de estudios de comunicación y ciencias sociales · doi:10.31921/doxacom.n32a5 · Cited by 7
“Los documentos sin conclusiones científicas se han desestimado y, a través del hipertexto, se aplicó metodología de investigación “bola de nieve” para localizar más documentos con las referencias de búsqueda seleccionadas en las mismas bases de datos (Wohlin, 2014).”
BackgroundOwoc, M. L., & Weichbroth, P. (2021) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-80847-1_5 · Cited by 2
A Systematic Review of Sustainable Supply Chain Management Practices in Food Industry
Minardi, F., Botta‐Genoulaz, V., & Mangano, G. (2021) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-85969-5_2 · Cited by 2
“Furthermore, a forward and backward snowballing procedure have been carried out to ensure that all valuable knowledge has been identified (Wohlin, 2014) also from the studies not identified through the initial search process 1 American Productivity and Quality Council (APQC).”
MethodA scoping review on the practices of open innovation
Guimarães, M. C., & Tortorella, G. L. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.649 · Cited by 2
“Snowball refers to using an article's reference list or article citations to identify additional articles [61].”
A Comparative Study of NLP based Semantic Web Standard model using SPARQL database
Rao, C. M., Babu, J. R., Pimo, S. J., et al. (2021) · doi:10.1109/iccs54944.2021.00010 · Cited by 2
ML@SE: What do we know about how Machine Learning impact Software Engineering practice?
Borges, O. T., Lima, M. H. T. D. F., Couto, J., Gadelha, B., Conte, T. U., & Prikladnicki, R. (2022) · Iberian Conference on Information Systems and Technologies (CISTI) · doi:10.23919/cisti54924.2022.9820309 · Cited by 2
A mobile application to protect groundwater during unconventional oil and gas extraction
Worthmann, C., & Esterhuyse, S. (2022) · Royal Society Open Science · doi:10.1098/rsos.220221 · Cited by 2
“Backward and forward snowballing was used to identify additional relevant literature sources from the articles that were identified from the databases [23].”
MethodIntensifying the search-based optimization of product line architectures with crossover operators
da Silva, D. F., Okada, L. F., Assunção, W. K. G., et al. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10198-3 · Cited by 2
“3 In addition to digital libraries, we also conducted manual searches on Google Scholar 4 and backwards snowballing (Wohlin 2014).”
MethodSuescún, E., Leite, J. C. S. D. P., & Pardo, C. (2022) · Revista Facultad de Ingeniería · doi:10.19053/01211129.v31.n60.2022.14189 · Cited by 3
“The process proposes to automate some steps of the Snowballing technique presented by Wohlin [26].”
BackgroundMethodInfluentialBelle, A. B., & Zhao, Y. (2022) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec57359.2022.00071 · Cited by 3
Automatic Data-Driven Software Change Identification via Code Representation Learning
Heričko, T. (2023) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3593434.3593505 · Cited by 7
“the automated search, we plan to employ snowballing as proposed byWohlin [32], which enables the identification of studies that may have been missed during the automated search.”
MethodDevelopment of a Capability Maturity Model for Organization-Wide Green IT Adoption
Brand, V., Razavian, M., & Özkan, B. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-36757-1_10 · Cited by 5
Ayyubi, S. E., Wahyuni, E. S., Muljono, P., & Beik, I. S. (2023) · Al-Muzara ah · doi:10.29244/jam.11.1.63-85 · Cited by 18
“Kemudian, dengan menggunakan set artikel awal ini, bola salju dilakukan mengikuti strategi yang diperkenalkan oleh Wohlin (2014).”
Background
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.