Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 2,201–2,300 of 3,934
Adaptive Reuse of Existing Buildings
Andreucci, M. B., & Karagözler, S. (2024) · Springer tracts in civil engineering · doi:10.1007/978-3-031-73490-8_11 · Cited by 5
Evans, R. S., McGregor, H. E., & Reed, B. (2024) · Education Citizenship and Social Justice · doi:10.1177/17461979241289280 · Cited by 10
“We used forward snowballing to search references of literature that we collected for relevant articles (Wohlin, 2014).”
Bena, Y. A., Ibrahim, R. B., Mahmood, J., Talpur, N., Nasser, M., Ayemowa, M. O., & Yusuf, M. N. (2024) · IEEE Access · doi:10.1109/access.2024.3498947 · Cited by 10
Pasqualotto, E., Pompeu, B. F., Braga, M. A. P., Delgado, L. M., Pedrotti, M., Ferreira, R. O. M., Pasqualotto, T., Morbach, V., & Formiga, F. B. (2024) · World Journal of Surgery · doi:10.1002/wjs.12443 · Cited by 8
Sangabriel-Alarcón, J., Ocharán-Hernández, J. O., Limón, X., & Cortés-Verdín, K. (2024) · Programming and Computer Software · doi:10.1134/s0361768824700749 · Cited by 10
“We chose the process proposed by Wohlim [25].”
The Value of Applying Big Data Analytics in Health Supply Chain Management
Nuaimi, D. A., & Awofeso, N. (2025) · F1000Research · doi:10.12688/f1000research.156525.4 · Cited by 5
Santana, A. C. B., Delgado, L. M., Guedes, L. S. D. S. P., Bossi, B. M., Leite, P. H. C. D. M., & Pompeu, B. F. (2025) · Journal of Robotic Surgery · doi:10.1007/s11701-025-02418-x · Cited by 6
“We also searched the references of the included studies and previous systematic reviews and meta-analyses aiming for the inclusion of additional studies [19].”
Gewehr, D. M., Carvalho, P. E. P., Martignoni, F., Nogueira, A., Gomes, C. P., Navalha, D. D. P., Pirez, J., Melo, L., de Mesquita, Y. L. L., Braga, M. A. P., et al. (2025) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwaf335 · Cited by 7
Designing for Behavior Change - 6 Dimensions of Social Comparison Features
Mylonopoulou, V., Väyrynen, K., & Isomursu, M. (2018) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2018.363 · Cited by 5
“Third, backwards snowball technique [29] was used to find more papers through the reference lists of the papers found.”
MethodBevan, G., Poli, C. D., Keng, M. J., & Raine, R. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-033483 · Cited by 5
de Oliveira, N. J. S., Procaci, T. B., & Siqueira, S. W. M. (2020) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2020.28.0.229 · Cited by 5
Mahl, T., Köhler, C., Arnold, D., Lins, D., & Kuhlenkötter, B. (2021) · Proceedings of the Design Society · doi:10.1017/pds.2021.511 · Cited by 6
“With “snowballing” (Wohlin, 2014), all titles from the bibliography of the selected papers were checked (backward snowballing) as well as all the papers citing the selected paper (forward snowballing).”
BackgroundSarancic, D., Pigosso, D. C. A., & McAloone, T. C. (2021) · Proceedings of the Design Society · doi:10.1017/pds.2021.454 · Cited by 11
“Additionally, the backward snowballing approach (Wohlin, 2014) was applied to include other relevant publications through cross-referencing.”
MethodTowards a Comprehensive Understanding of Agile Teamwork: A literature-based Thematic Network
Freire, A. (2021) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2021-106 · Cited by 6
“Wohlin [19] recommends Google Scholar to avoid bias in favor of any specific publisher.”
BackgroundMethodOn Material selection and its consequences in product development
Henriksson, F. (2021) · Linköping studies in science and technology. Dissertations · doi:10.3384/diss.diva-178363 · Cited by 5
Knowledge-based Risk Management: A Systematic Literature Review
Neto, A. F. D. S., Perkusich, M., Dantas, E., Ramos, F., Costa, A., de Almeida, H. O., & Perkusich, Â. (2021) · Brazilian Symposium on Software Engineering · doi:10.1145/3474624.3474635 · Cited by 10
A Comprehensive Survey on Big Data Technology Based Cybersecurity Analytics Systems
Saravanan, S., & Prakash, G. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-33-6173-7_9 · Cited by 5
Jaumot‐Pascual, N., Silva, C. B., Martínez-Gudapakkam, A., & Ong, M. (2021) · Research on Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT) · doi:10.1109/respect51740.2021.9620675 · Cited by 8
“Forward and backward snowballing [17] were used to identify additional literature.”
MethodToward a Maturity Model for Digital Business Ecosystems from an IT perspective
Ehrensperger, R., Sauerwein, C., & Breu, R. (2021) · IEEE International Enterprise Distributed Object Computing Conference (EDOC) · doi:10.1109/edoc52215.2021.00012 · Cited by 10
A Literature-Based Thematic Network to Provide a Comprehensive Understanding of Agile Teamwork (106)
Freire, A., Neto, M., Perkusich, M., Costa, A. A., Gorgônio, K., de Almeida, H. O., & Perkusich, Â. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500176 · Cited by 6
Benefits and Challenges of a Graduate Course: An Experience Teaching Systematic Literature Review
Iwazaki, A. Y., dos Santos, V., Felizardo, K. R., de Souza, É. F., Valentim, N., & Nakagawa, E. Y. (2022) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie56618.2022.9962446 · Cited by 6
Successful management of cloud‐based global software development projects: A multivocal study
Akbar, M. A., Khan, A. A., Mahmood, S., & Smolander, K. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2527 · Cited by 9
“”The snowballing is an effective way to collect the most potential literature related to the context of the study[29, 30].”
BackgroundIslamic Educational Thought: Zakiyah Darajat's Perspective
Nasikhin, N., Ghofur, A., Rokhmadi, R., et al. (2022) · Al-Aulia Jurnal Pendidikan dan Ilmu-Ilmu Keislaman · doi:10.46963/aulia.v8i2.706 · Cited by 5
On the assignment of commits to releases
Pinto, F. C. D. R., & Murta, L. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10263-x · Cited by 8
“We conducted our systematic literature review using the snowballing approach and the guidelines proposed by Wohlin (2014).”
MethodCfgNet: A Framework for Tracking Equality-Based Configuration Dependencies Across a Software Project
Simon, S., Ruckel, N., & Siegmund, N. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3274349 · Cited by 5
“In the next step, we extended our initial set of relevant papers, following a forward and backward snowballing approach [27].”
MethodUsing RFID in the Engineering of Interactive Software Systems: A Systematic Mapping
Maia, V. C., de Oliveira, K. M., Kolski, C., & Travassos, G. H. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3593235 · Cited by 6
“The execution of the search string was followed by snowballing , a systematic literature review method that permits the selection of additional papers through a reference list of works or citations [108].”
MethodOkudan, O., Cevikbas, M., & Işık, Z. (2023) · Engineering Construction and Architectural Management · doi:10.1108/ecam-02-2023-0105 · Cited by 5
Towards an E-Government Enterprise Architecture Framework for Developing Economies
Namagembe, F., Nakakawa, A., Tulinayo, F. P., Proper, H. A., & Overbeek, S. J. (2023) · Complex Systems Informatics and Modeling Quarterly · doi:10.7250/csimq.2023-35.02 · Cited by 9
Tech4Bad in the Oil and Gas Industry: Exploring Choices for ICT Professionals
Brooks, I., Thorslund, M. L., & Biørn-Hansen, A. (2023) · International Conference on ICT for Sustainability (ICT4S) · doi:10.1109/ict4s58814.2023.00023 · Cited by 11
“…we performed searches in English on Google as the world’s largest search engine and Twitter as a platform frequently used by social movements to 1) systematically document the background, cause, organisers and targets of each case; and 2) find further initiatives through snowball sampling [18].”
MethodContract Testing in Microservices-Based Systems: A Survey
Simosa, M. L., & Siqueira, F. (2023) · IEEE International Conference on Software Engineering and Service Science (ICSESS) · doi:10.1109/icsess58500.2023.10293058 · Cited by 8
“Finally, a snowball process was executed, which consists of searching for related and important studies through their references (snowball backwards) and works that cite studies (snowball forward) [9].”
MethodA Systematic Mapping Study of Digital Twins for Diagnosis in Transportation
Prikler, L. M., & Wotawa, F. (2023) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa59317.2023.00058 · Cited by 11
“We applied backward and forward snowballing as outlined by Wohlin [19] with a preference towards using COCI [20] and CROCI [21] to quickly retrieve DOIs whenever possible.”
MethodResilient and sustainable production systems: towards a research agenda
Säfsten, K., Harlin, U., Johansen, K., Larsson, L., & Rönnbäck, A. Ö. (2023) · International Journal of Manufacturing Research · doi:10.1504/ijmr.2023.135651 · Cited by 5
A Tertiary Study on Microservices: Research Trends and Recommendations
Stojanov, Ž., Hristoski, I. S., Stojanov, J., & Stojkov, A. (2023) · Programming and Computer Software · doi:10.1134/s0361768823080200 · Cited by 6
“…selecting the most recent version of the study (if there are multiple versions by the same authors); (cid:129) Ph#3: Employing the snowball method [30] to search for additional studies and minimize the possibility of overlooking or omitting any relevant secondary studies; (cid:129) Ph#4: Applying…”
How to Support ML End-User Programmers through a Conversational Agent
Garcia, E. J. A., Pimentel, J. F., Feng, Z., Gerosa, M. A., Steinmacher, I., & Sarma, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608130 · Cited by 19
Cybersecurity Risk Audit: A Systematic Literature Review
Sánchez-García, I. D., Rea-Guaman, Á. M., Feliú, T. S., & Calvo-Manzano, J. A. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-50590-4_18 · Cited by 9
You cannot improve what you do not measure: A triangulation study of software security metrics
Kudriavtseva, A., & Gadyatskaya, O. (2024) · ACM Symposium on Applied Computing · doi:10.1145/3605098.3635892 · Cited by 10
“Then, using the snowballing method [75], we extended the set of metrics with an additional literature study to make sure that important metrics, especially those proposed in the last few years, were not omitted.”
MethodPursuing sustained competitive advantage through the use of process management
Navarro, P., & Haag, L. (2024) · International Journal of Lean Six Sigma · doi:10.1108/ijlss-12-2023-0219 · Cited by 5
Reporting case studies in systematic literature studies—An evidential problem
Rainer, A., & Wohlin, C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107501 · Cited by 7
Self-citationEnhancing AI Education Through Educational Games: Evidence From a Meta-Analysis Study
Lu, Y., Lan, J., & Li, K. (2024) · International Conference on Educational and Information Technology · doi:10.1109/iceit61397.2024.10540611 · Cited by 8
“Snowballing is a method of obtaining additional literature by using references and citations from existing literature and providing additional information to a meta-analysis dataset[14].”
MethodFitzgerald, N., O’Donnell, R., Uny, I., et al. (2024) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2024.104465 · Cited by 5
Intersection of machine learning and mobile crowdsourcing: a systematic topic-driven review
Chen, W., Hussain, W., Al-Qudah, I., Al‐Naymat, G., & Zhang, X. (2024) · Personal and Ubiquitous Computing · doi:10.1007/s00779-024-01820-w · Cited by 7
“Note that to ensure the thoroughness of the review, a snowballing technique [10] has been applied where necessary.”
Towards engineering higher quality intelligent environments: a multi case study approach
Santokhee, A., Augusto, J. C., & Brodie, L. (2024) · Software Quality Journal · doi:10.1007/s11219-024-09678-0 · Cited by 6
Artificial intelligence applications in auditing processes in the banking sector
Albahsh, R., & Al-Anaswah, M. F. (2024) · Corporate Ownership and Control · doi:10.22495/cocv21i3art3 · Cited by 5
“Additionally, a forward snowballing sampling technique was employed on the most cited papers, resulting in the inclusion of six more studies (Wohlin, 2014).”
MethodSharing Software-Evolution Datasets: Practices, Challenges, and Recommendations
Broneske, D., Kittan, S., & Krüger, J. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660798 · Cited by 5
“…on guidelines and recommendations for searching papers for systematic literature reviews and mapping studies in software engineering [Brereton et al., 2007, Jalali and Wohlin, 2012, Kitchenham et al., 2015, Kitchenham and Charters, 2007, Krüger et al., 2020, Shakeel et al., 2018, Wohlin, 2014].”
Sachete, A. D. S., Gomes, R. S., Filho, A. B. D. C., et al. (2024) · RELATEC Revista Latinoamericana de Tecnología Educativa · doi:10.17398/1695-288x.23.2.69 · Cited by 5
“(3) We do not adopt Snowballing (Wohlin, 2014), a technique that allows the inclusion of references from accepted articles in the research corpus that meet the inclusion and exclusion criteria.”
MethodBuck, D., Mulligan, L. D., Lennox, C., Bowden, J., Minchin, M., Kemp, L., Devine, L., Southworth, J., Ghafur, F., Robinson, C., et al. (2024) · Medicine Science and the Law · doi:10.1177/00258024241264762 · Cited by 8
“Snowballing methods 34 were used to identify additional sources.”
Participatory and Collaborative Modeling of Sustainable Systems: A Systematic Review
Manellanga, R. A., & Dávid, I. (2024) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3652620.3688557 · Cited by 12
“Backward snowballing is conducted by the recommendations of Wohlin [85].”
Digital risk: a systematic multivocal literature review
Ramalho, D., Fernandes, A. D., da Silva, M. M., & Pereira, R. (2024) · Information Security Journal A Global Perspective · doi:10.1080/19393555.2024.2419920 · Cited by 6
“Two additions were made to the search criteria during result collection: • Due to the low number of accepted results yielded from “allintitle:‘digital risk management’” (two from ADBs, six from Google), the authors decided to attempt a backwards snowballing (Wohlin, 2014) of the Google results.”
Participating in Rural Appraisal? A Review of Participatory GIS and Citizen Science Approaches
Ntassiou, K. (2025) · Geography Compass · doi:10.1111/gec3.70023 · Cited by 5
Requirements-Driven Automated Software Testing: A Systematic Review
Wang, F., Arora, C., Tantithamthavorn, C. K., Huang, K., & Aleti, A. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3767739 · Cited by 21
Axioms of Linguistic Architecture
Heinz, M., Lämmel, R., & Varanovich, A. (2017) · International Conference on Model-Driven Engineering and Software Development · doi:10.5220/0006210204780486 · Cited by 5
“We did not perform snowballing (Wohlin, 2014) to limit the amount of papers, as the analysis for paper inclusion is relatively laborious.”
MethodDagonneau, J., Rocks, S. A., Prpich, G. P., Garnett, K., Black, E., & Pollard, S. (2017) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2017.03.293 · Cited by 6
“Additional sources were identified from relevant document bibliographies using a snowballing technique (Wohlin, 2014).”
BackgroundOrdering the Product Backlog in Agile Software Development Projects: A Systematic Literature Review
Silva, A., Silva, A., Araújo, T., Barbosa, R., Ramos, F., Costa, A. A., Perkusich, M., & Dilorenzo, E. (2017) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2017-007 · Cited by 11
Value-Based Requirements Engineering: Challenges and Opportunities
Wnuk, K., & Mudduluru, P. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-99617-2_2 · Cited by 5
Approaching location-based services from a place-based perspective: from data to services?
Bahrehdar, A. R., Koblet, O., & Purves, R. S. (2019) · Journal of Location Based Services · doi:10.1080/17489725.2018.1564383 · Cited by 4
“To select a broad range of relative literature we used a combination of purposive and snowball sampling (Wohlin 2014).”
MethodA comparison and evaluation of variants in the coupling between objects metric
Child, M., Rosner, P., & Counsell, S. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.02.020 · Cited by 8
“A basic snowballing technique was then used to recursively follow both the citations and references given for, all the relevant papers identified [Wohlin2014; Brereton2007].”
MethodUsing Awareness Information to Enhance Online Discussion Forums: A Systematic Mapping Study
de Lima, D. P. R., Gerosa, M. A., & Netto, J. F. D. M. (2018) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2018.8658487 · Cited by 5
“In addition, a manual search was also conducted following the snowballing approach as defined by Wohlin [5].”
MethodThe Theory and Practice of Educational Data Forensics
de Klerk, S., van Noord, S., & van Ommering, C. J. (2019) · Methodology of educational measurement and assessment · doi:10.1007/978-3-030-18480-3_20 · Cited by 5
“Because the literature search did not yield the desired results, a snowballing approach, presented by Wohlin (2014), was used to find more relevant literature on the topic of test security.”
BackgroundInfluentialLet’s Talk About Tools and Approaches for Teaching HCI
Lima, A. L. D. S., & Benitti, F. B. V. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-21814-0_13 · Cited by 6
Experiência do Usuário em Jogos Digitais: Uma Catalogação de Instrumentos de Avaliação
Borges, B., Sidarta, I., de Souza, A. M., Coelho, B., & Darin, T. (2019) · doi:10.5753/wiplay.2019.7841 · Cited by 7
Monitoring and Understanding Enterprise Collaboration Platform Outcomes and Benefits Change
Nitschke, C. S., & Williams, S. P. (2020) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2020.318 · Cited by 6
“The search process was extended by backward and forward snowball sampling [93] to identify further potentially relevant literature.”
MethodIdentifying PLM themes and clusters from a decade of research literature
Maranzana, N., Segonds, F., Rivest, L., Braesch, C., Nyffenegger, F., & Danjou, C. (2019) · International Journal of Product Lifecycle Management · doi:10.1504/ijplm.2019.107005 · Cited by 12
Ethical Considerations in Case Studies
Yu, L. (2020) · doi:10.1109/rethics51204.2020.00009 · Cited by 8
“In this report, 21 ethical considerations have been extracted through the literature review [24].”
BackgroundMethodVariability Modeling in Software Product Line: A Systematic Literature Review
Jaffari, A., Lee, J., & Kim, E. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-64773-5_1 · Cited by 6
Architecture enforcement concerns and activities - An expert study
Schröder, S., Soliman, M., & Riebisch, M. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.08.025 · Cited by 10
Concepts and Models of Environment of Self-Adaptive Systems: A Systematic Literature Review
Shin, Y.-J., Bae, J.-Y., & Bae, D. (2021) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec53868.2021.00037 · Cited by 6
“2The snowballing method exhaustively explores all the backward references (cited by the subject paper) and forward references (citing the subject paper) until no additional papers are discovered [16].”
MethodConstruction transport efficiency from the perspective of Main Contractor and Transporter
Naz, F. (2022) · Linköping studies in science and technology. Thesis · doi:10.3384/9789179292744 · Cited by 6
Mehta, A., Mehta, N., & Bindal, I. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111345 · Cited by 12
Smart Indoor Space Simulation Methodologies: A Review
Golestan, S., Stroulia, E., & Nikolaidis, I. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3159205 · Cited by 6
Evaluating and Evolving a Conceptual Model for Projects Observatories
Vieira, J. K. M., de Farias, I. H., & de Moura, H. P. (2022) · doi:10.1145/3535511.3535540 · Cited by 5
“Os guidelines de [8, 27] para o desenvolvimento de pesquisas de re-visão/mapeamento foram seguidos nessa etapa.”
BackgroundBetween Caste and Occupation: Issues of Sweeper Community in Kashmir
Ahmad, K. M., & Shah, H. (2022) · Contemporary Voice of Dalit · doi:10.1177/2455328x221108286 · Cited by 7
Generated Tests in the Context of Maintenance Tasks: A Series of Empirical Studies
Herculano, W. B. R., Alves, E. L. G., & Mongiovi, M. (2022) · IEEE Access · doi:10.1109/access.2022.3222803 · Cited by 6
“For this study, we recruited volunteers by convenience (contacting developers from partner companies and universities), social networking platforms (e.g., LinkedIn), and snowball sampling [45], i.e., participants were asked to resend the survey invitation to others.”
A Systematic Review on the Application of Educational Robotics to Children with Learning Disability
da Silva, M. D. G. T., Albuquerque, E. A. Y., & Gonçalves, L. M. G. (2022) · Latin American Robotics Symposium, Brazilian Symposium on Robotics and Workshop on Robotics in Education (LARS/SBR/WRE) · doi:10.1109/lars/sbr/wre56824.2022.9995956 · Cited by 5
“There are other approaches for determining the state of the art in some area or topic, as the snowballing [30].”
BackgroundA Comprehensive Approach for Applying Threat Modeling to Internet of Things Systems
da Silva, L. P., Nascimento, B. S., Dias, R. A. M. P., & Mendonça, D. (2022) · IEEE World Forum on Internet of Things (WF-IoT) · doi:10.1109/wf-iot54382.2022.10152291 · Cited by 10
“Finally, we performed forward snowballing [16] on the remaining papers, getting the final list with ten surveys to analyze.”
MethodMatraf, M. S. B., Hashim, N. L., & Hussain, A. (2023) · Journal of Information and Communication Technology · doi:10.32890/jict2023.22.3.5 · Cited by 8
Clear, T. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2835019 · Cited by 8
“There are other approaches to being systematic in the conduct of reviews [19, 1], which could equally be considered.”
BackgroundMultaharju, S. (2016) · Operations and Supply Chain Management An International Journal · doi:10.31387/oscm0250171 · Cited by 6
“Next, in order to ensure the achievement of all the relevant articles into the shortlist, we conducted both backward and forward snowballing procedures (Wohlin, 2014).”
MethodA Systematic Mapping of Literature on Software Refactoring Tools
Tavares, C. S., Ferreira, F., & Figueiredo, E. (2018) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3229345.3229357 · Cited by 7
“Por fim, utilizou-se a técnica de Bola de Neve (Snowballing)[49] para incluir trabalhos que os artigos selecionados referenciam.”
What is the role of accounting in disaster recovery and relief? A literature review
Fahlevi, H., Indriani, M., Mulyany, R., et al. (2019) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/273/1/012059 · Cited by 5
Software Engineering Education Beyond the Technical: A Systematic Literature Review
Groeneveld, W., Vennekens, J., & Aerts, K. (2019) · arXiv.org · doi:10.48550/arxiv.1910.09865 · Cited by 29
“A technique called ‘snowballing’ was adopted to consider publications from reference lists of papers in the review pool [16].”
MethodInfluentialCharacterization of software testing practices: A replicated survey in Costa Rica
Quesada–López, C., Hernández–Agüero, E., & Jenkins, M. (2019) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2019.472 · Cited by 5
“Additionally, we applied the snowballing technique ( Wohlin, 2014) on two surveys previously published (Garousi and Zhi, 2013; Dias-Neto et al., 2017).”
MethodTowards a Roadmap for the Internet of Things Software Systems Engineering
Motta, R. C., de Oliveira, K. M., & Travassos, G. H. (2020) · International Conference on Management of Emergent Digital EcoSystems (MEDES) · doi:10.1145/3415958.3433100 · Cited by 6
“We used Scopus as the search engine since, incremented the search with Snowballing procedures (backward and forward) [16] .”
MethodNahar, K., Gill, A. Q., & Roach, T. (2021) · Security and Privacy · doi:10.1002/spy2.160 · Cited by 6
“In order to study existing research, a keyword-based search was performed in various research databases coupled with the snowballing technique of literature review 61 .”
MethodRevisões Sistemáticas da Literatura: Parte II
Ruschel, R. C., Neves, L. D. O., Bernardini, S. P., & Moreira, D. D. C. (2017) · PARC Pesquisa em Arquitetura e Construção · doi:10.20396/parc.v8i4.8652036 · Cited by 6
O Ensino de Programação com Scratch e seu Impacto na Opção Profissional para Meninas
de Mattos, F., Ferreira, V., & Anacleto, J. C. (2016) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/cbie.sbie.2016.300 · Cited by 5
“Para complementar o processo de busca automático nas bases definidas no estudo piloto, foi adotado o método de snowballing como estratégia para buscar recursivamente publicações que não foram recuperadas no processo de busca, mas que podem ser interessantes para o estudo [Wohlin 2014].”
BackgroundSetting the Scope for a New Agile Assessment Model: Results of an Empirical Study
Tuncel, D., Körner, C., & Plösch, R. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-78098-2_4 · Cited by 9
“Towards this goal, we have initiated our research with a Systematic Literature Review based on backwards and forward snow-balling described by Wohlin [11], the systematic literature review study allowed us identifying reusable components of the previously conducted researches, and the already established assessment mo…”
MethodMunkácsy, A. (2017) · UPM Digital Archive (Technical University of Madrid) · doi:10.20868/upm.thesis.47849 · Cited by 8
Time Pressure in Software Engineering: A Systematic Literature Review.Preprint
Kuutila, M., Mäntylä, M., Farooq, U., et al. (2019) · arXiv (Cornell University) · Cited by 5
Model-driven Engineering of Safety and Security Systems: A Systematic Mapping StudyPreprint
Mashkoor, A., Egyed, A. F., & Wille, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08471 · Cited by 5
“In a snowballing reading, the lists of references of found papers are used to identify other relevant studies [110].”
MethodYedida, R., Yang, X., & Menzies, T. (2021) · arXiv (Cornell University) · Cited by 4
Self-Sovereign Identity: A Systematic Map and Review.Preprint
Schardong, F., & Custódio, R. F. (2021) · arXiv (Cornell University) · Cited by 6
Cybersecurity Of Work From Anywhere Model For Government : A Systematic Literature Review
Asyrofi, M. F., & Nugraha, I. G. D. (2025) · International Journal of Electrical, Computer, and Biomedical Engineering · doi:10.62146/ijecbe.v3i1.113 · Cited by 5
Managing and Controlling Digital Role-Playing Game Elements: A Current State of Affairs
da Rocha Franco, A. D. O., de Carvalho, W. V., da Silva, J. W. F., Maia, J. G. R., & de Castro, M. F. (2024) · Entertainment Computing · doi:10.1016/j.entcom.2024.100708 · Cited by 5
From Trust to Appropriate Reliance: Measurement Constructs in Human-AI Decision-Making
Raees, M., & Papangelis, K. (2026) · arXiv · doi:10.48550/arxiv.2604.23896 · Cited by 5
“In another iteration, we looked at the relevant references of included studies to identify more papers using snowballing [93] (by reviewing their titles and abstracts); however, no new results matching the criteria were found.”
Cryptocurrency Trading: A Comprehensive Survey
Fang, F., Ventre, C., Basios, M., Kanthan, L., Martínez‐Rego, D., Wu, F., & Li, L. (2025) · Springer eBooks · doi:10.1007/978-981-96-6839-7_4 · Cited by 9
Iqbal, J., Jibran, H., Al‐Shamayleh, A. S., Abbas, F., Akhunzada, A., Alharthi, S. Z., & Gani, A. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1656 · Cited by 5
“In the next step, we carried out the snowballing method ( Wohlin, 2014 ) to check the references of the 13 papers.”
MethodCucchi, M., Mariani, S., Kawczynski, M. J., Shkurka, E., Ius, F., Comentale, G., Hoskote, A. U., & Lorusso, R. (2024) · Perfusion · doi:10.1177/02676591241240377 · Cited by 4
Setioningtyas, W. P., Fodor, Z. J., & Dunay, A. (2024) · Cogent Business & Management · doi:10.1080/23311975.2024.2396534 · Cited by 7
“…note that these two processes were executed as distinct and independent phases, aligning with established methodologies as detailed in the works of wohlin (2014), and Mourao et al. (2017). this approach was thoughtfully selected to enhance the precision and comprehensiveness of our search for…”
The Contribution of RFID to Circular Economy Deployment: A Literature Review
Gallo, M., Viola, A., & Romagnoli, G. (2025) · International Journal of RF Technologies · doi:10.1177/17545730251331373 · Cited by 8
“The snowball search method is a method to fi nd other relevant publications that are included in the reference list of the seed paper, aka backward snowballing, or that cite the seed paper itself, aka forward snowballing (Wohlin, 2014).”
Liao, X., Kamsin, A., & Noor, N. F. B. M. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2542898 · Cited by 8
“The snowballing method was utilized, which involves retrieving relevant papers from the reference lists of target papers or papers citing them (Wohlin, 2014).”
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.