Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 2,501–2,600 of 3,934
Mapping supply systems in circular settings: A proposal for a new methodology
Zerbino, P., Mabroum, S., Montalbano, G., Lima, A. T., & Aloini, D. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.110891 · Cited by 4
Aliska, I., Knudsen, S., Mehdi, Z., & Anson, S. (2025) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2025.105214 · Cited by 6
Drivers and challenges for wood-based construction in urban areas
Iliescu, O., Amiri, A., & Junnila, S. (2025) · Environmental Research Communications · doi:10.1088/2515-7620/adaf0f · Cited by 6
Exploring the Environment of Technology Adoption: Challenges and Effective Strategies
Ali, O., & Al‐Ahmad, A. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.02.042 · Cited by 5
A Mapping Review to Understand Web and Mobile Apps Accessibility for Adults with Autism
Ribeiro, D. M., Azul, C., Negromonte, V., Melo, F. V., Matias, G. W., Farias, A., Chaves, A. P., & da Gama, K. S. (2025) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi.2025.246487 · Cited by 4
Zoghi, A., Bilodeau, É., Khaliq, M. N., Kim, Y., Martel, J., & Drake, J. A. P. (2025) · Journal of Hydrology Regional Studies · doi:10.1016/j.ejrh.2025.102460 · Cited by 17
A survey of coverage-guided greybox fuzzing with deep neural models
Qiu, J., Jiang, Y., Miao, Y., Luo, W., Pan, L., & Zheng, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107797 · Cited by 8
Bolzan, W., & Farias, K. (2025) · ACM Computing Surveys · doi:10.1145/3742899 · Cited by 7
“To mitigate this threat, they add research to the Snowballing method [33, 74], [39], in which the articles selected in the last step were used as a starting set for the iteration of their references.”
León, S. M., Robles, G., & Gonzalez-Barahona, J. M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112628 · Cited by 3
Should We Adopt a New Version of a Standard? – A Method and Its Evaluation on AUTOSAR
Motta, C., Durisic, D., & Staron, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49094-6_9 · Cited by 3
“First, we conducted a literature review on the existing approaches for monitoring the evolution of system requirements using the snowball method [16].”
MethodMesquita, L., & Sánchez, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23563-5_22 · Cited by 3
Modularity in the Construction Industry: A Systematic Mapping Study
Bridi, M. E., Ceolin, E. D., Granja, A. D., et al. (2019) · Annual Conference of the International Group for Lean Construction · doi:10.24928/2019/0178 · Cited by 3
The Educators Personality Development in Industry 4.0 Based on Pumping HR-Model
Ramly, A. T., & Aminda, R. S. (2020) · IJBE (Integrated Journal of Business and Economics) · doi:10.33019/ijbe.v4i2.258 · Cited by 3
The Decision Arena: A model-centric interactive workspace for Product-Service System design
Wall, J., & Bertoni, M. (2020) · doi:10.35199/norddesign2020.37 · Cited by 3
“In the DS-I, relevant contributions were collected through a systematic search in the Scopus and Google Scholar databases and further complemented through snowballing (Wohlin, 2014) to find additional literature.”
BackgroundBest Practices for Software Development: A Systematic Literature Review
Ordoñez-Pacheco, R., Cortés-Verdín, K., & Ocharán-Hernández, J. O. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-63329-5_3 · Cited by 4
Rosa, V., & Neto, J. C. (2020) · Research Society and Development · doi:10.33448/rsd-v9i10.9019 · Cited by 10
Guidelines for braindraw conduction: looking at braindraw under the lens of abduction and semiotics
Rosa, J. C. S., Valente, P. D., Costa, L., Nunes, N., & Matos, E. D. S. (2021) · Behaviour and Information Technology · doi:10.1080/0144929x.2021.1925742 · Cited by 3
“We also carried out forward snowballing (Wohlin 2014) searching for new articles based on the found articles ’ citations.”
MethodInnovative Inquiry-based Methods in Learning and Teaching Science
Bónus, L., & Antal, E. (2021) · Journal of Studies in Education · doi:10.5296/jse.v11i3.18700 · Cited by 9
“Additional articles were identified using a snowballing method (Wohlin, 2014).”
MethodModel for Projects Observatories: A Preliminary Study
Vieira, J. K. M., de Farias, I. H., & de Moura, H. P. (2021) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3466933.3466940 · Cited by 3
“Essa etapa foi realizada seguindo as diretrizes apresentadas por [38] para o uso de snowballing em pesquisas em Engenharia de Software.”
BackgroundIntroducing adaptive mechatronic designs in bulk handling industry
van Kol, T. V., Jovanova, J., Mohajeri, M. J., et al. (2021) · IEEE/ASME International Conference on Advanced Intelligent Mechatronics · doi:10.1109/aim46487.2021.9517377 · Cited by 3
“Additional information regarding snowballing methods is provided by [29].”
MethodPesqueira, A. M. (2021) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-030-75123-4_1 · Cited by 3
Measurement of interpersonal trust in virtual software teams: A systematic literature review
Zapata, S., Barros‐Justo, J. L., Matturro, G., & Sepúlveda, S. (2021) · Ingeniare. Revista chilena de ingeniería · doi:10.4067/s0718-33052021000400788 · Cited by 4
“Discrepancies over the qualification of each study between researchers were resolved following the guidelines from Wohlin [37].”
MethodInfluentialSystematizing the Meta-Analytical Process in Software Engineering
Jaccheri, L., Kholmatova, Z., & Succi, G. (2021) · European Symposium on Software Engineering (ESSE) · doi:10.1145/3501774.3501775 · Cited by 6
“which it is so and to make the first attempts to systematize the process of aggregation of the results we started with analyzing the existing literature in this field using PICO approach [19] and snowballing techniques [20].”
MethodSelbstbestimmung, Privatheit und Datenschutz: Gestaltungsoptionen für einen europäischen Weg
Friedewald, M., Kreutzer, M., & Hansen, M. (2022) · DuD-Fachbeiträge · doi:10.1007/978-3-658-33306-5 · Cited by 6
de Carvalho, E. A., Gomes, J. O., Jatobá, A., da Silva, M. F., & de Carvalho, P. V. R. (2022) · iSys - Brazilian Journal of Information Systems · doi:10.5753/isys.2022.2220 · Cited by 4
“Nessa abordagem, geralmente, o primeiro desafio é identificar um conjunto inicial de artigos para iniciar o procedimento [Wohlin 2014].”
BackgroundEnergy inefficiency diagnosis for Android applications: a literature review
Sun, Y., Fang, J., Chen, Y., et al. (2022) · Frontiers of Computer Science · doi:10.1007/s11704-021-0532-4 · Cited by 3
Code Smell Co-occurrences: A Systematic Mapping
Neto, A., Bezerra, C. I. M., & Martins, J. S. (2022) · Brazilian Symposium on Software Engineering · doi:10.1145/3555228.3555268 · Cited by 4
“To complement the SM, we use backward snowballing, which means using the reference list to identify new documents to be included [25].”
MethodDeveloping research questions in conversation with the literature: operationalization & tool support
Díaz, Ó., & Contell, J. P. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10204-8 · Cited by 8
Ali, A., Pandya, S., & VARMA, U. C. (2022) · International journal of doctoral studies · doi:10.28945/5026 · Cited by 9
Mårtensson, T., Ancher, G., & Ståhl, D. (2023) · Software Testing Verification and Reliability · doi:10.1002/stvr.1839 · Cited by 3
“With this strict limitation on the number of iterations and number of additional searches, the literature review was conducted following the guidelines for literature studies from Wohlin [15]:”
InfluentialThe Debt and Non-Debt Tax Shields Trade-Off: A Review
Gabrielli, A. (2023) · International Journal of Academic Research in Accounting Finance and Management Sciences · doi:10.6007/ijarafms/v13-i1/16170 · Cited by 9
Kudo, T. N., Bulcão-Neto, R. F., Vincenzi, A. M. R., de Souza, É. F., & Felizardo, K. R. (2022) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2021.1978 · Cited by 3
Vichos, T., Rezaie, A. R., Vichos, P., Cash, B. D., & Pimentel, M. (2023) · Digestive Diseases and Sciences · doi:10.1007/s10620-023-07885-6 · Cited by 5
“These articles were screened, using the same methodology, for potentially relevant papers, as were the references of additional eligible studies (“snowball method”) [12].”
MethodLearning Theories in Games That Teach Responsible Antibiotic Use: A Literature Review
Molnar, A., & Molnar, I. (2023) · Sustainability · doi:10.3390/su15054643 · Cited by 5
Pamparău, C. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140488 · Cited by 7
“After the eligibility stage, we arrived at a number of 39 relevant papers, for which we applied two snowballing procedures [12]: (1) backward snowballing , where we analyzed the references of all the eligible papers, total of 2014 papers, and (2) forward snowballing , where we analyzed their Google…”
MethodSilitonga, R., & Simatupang, P. (2023) · Pharos Journal of Theology · doi:10.46222/pharosjot.104.322 · Cited by 3
“The literature search was conducted using the snowballing technique and relevant articles and books etc were used (Wohlin, 2014:321-330).”
MethodMorris, V. S., Ghazi, H., Fletcher, D. M., & Guinn, B. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24119667 · Cited by 4
"Nutritional Status of Formal and Informal Caregivers: A Systematic Review"
Teixeira, B. (2023) · Biomedical Journal of Scientific & Technical Research · doi:10.26717/bjstr.2023.50.007913 · Cited by 5
ADAPTING A DASHBOARD-BASED APPROACH FOR FEASIBILITY ANALYSIS TO CIRCULAR PSS BUSINESS MODELS
Mahl, T., Petry, M., & Köhler, C. (2023) · Proceedings of the Design Society · doi:10.1017/pds.2023.47 · Cited by 3
“To extend the range of the literature review the "snowballing"-method by Wohlin (2014) was used.”
MethodSuryani, N. P., Agustina, H. R., Eriyani, T., & Setyorini, D. (2023) · Jurnal Aisyah Jurnal Ilmu Kesehatan · doi:10.30604/jika.v8i3.2005 · Cited by 3
“The author also conducted an article search using a manual search technique to find grey literature through search engines such as Google Scholar and nursing organization websites and adapted the snowball technique to identify additional sources following the used references and recommended sources (Jaskiewicz & Tulen…”
Ter‐Minassian, M., DiNucci, A. J., Barrie, I., Schoeplein, R., Chakravarty, A. G., & Hernández‐Muñoz, J. J. (2023) · Annals of Epidemiology · doi:10.1016/j.annepidem.2023.07.006 · Cited by 3
Towards semantic operationalisation of the business model: a step forward
Milei, P., & Barajas, Á. (2023) · Management Research The Journal of the Iberoamerican Academy of Management · doi:10.1108/mrjiam-11-2022-1362 · Cited by 3
A ÁREA 46 NA CAPES: ORIGEM, MUDANÇAS E CONSOLIDAÇÃO COMO “ENSINO” NO CAMPO ACADÊMICO-CIENTÍFICO
Agostini, G., & Massi, L. (2023) · Investigações em Ensino de Ciências · doi:10.22600/1518-8795.ienci2023v28n2p65 · Cited by 3
dos Santos, R. N., Cunha, F., Rique, T. P., Perkusich, M., Neto, A. F. D. S., Albuquerque, D. W., de Almeida, H. O., & Perkusich, Â. (2023) · Brazilian Symposium on Software Engineering · doi:10.1145/3613372.3613404 · Cited by 4
“Following the guidelines outlined by Wohlin et al. [42], forward snowballing is an iterative process that involves identifying relevant studies by examining the citations of a seed set of studies.”
MethodSoftware Merge: A Two-Decade Systematic Mapping Study
Carbonera, C. E., Farias, K., Graeff, C. A., et al. (2023) · doi:10.1145/3613372.3613391 · Cited by 3
“We added eight studies, after following the guidelines defined in [22].”
MethodTowards Power Relationship Dynamics and Community Smells in the Proprietary Software Ecosystem
Ribeiro, B. B., Costa, L. A., do Outão, J. C. S., et al. (2023) · doi:10.1145/3613372.3613374 · Cited by 3
Hamada, R., & Kuzminykh, I. (2023) · Signals · doi:10.3390/signals4040038 · Cited by 6
“According to Wohlin’s [17] description, backward snowball sampling was employed to account for any important documents that might not have been found during the primary and secondary searches.”
Software Testing applied to the Development of IoT Systems: preliminary results
Ferreira, V. G., Herrera, C. G., Souza, S. R. S., et al. (2023) · doi:10.1145/3624032.3624049 · Cited by 3
“This method helps map the state of the art, identify knowledge gaps, find influential authors and research groups, and evaluate the quality and impact of publications [98].”
MethodPrivacy-Compliant Software Reuse in Early Development Phases: A Systematic Literature Review
Guber, J., & Reinhartz-Berger, I. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107351 · Cited by 10
Planejamento para o imprevisto em projetos de construção: uma revisão
do Carmo, C. S. T., & Sotelino, E. D. (2023) · Brazilian Journal of Production Engineering · doi:10.47456/bjpe.v9i4.42244 · Cited by 3
A comprehensive database of squirrel distribution and occurrence in South Asia
Udayraj, S., D’Souza, S., Aravind, P., Muni, R. K., & Rajamani, N. (2023) · Biodiversity Data Journal · doi:10.3897/bdj.11.e109946 · Cited by 3
Digital Revolution in Higher Education in the Covid-19 and Post Covid-19 Era
Dlamini, R. S. (2023) · Springer eBooks · doi:10.1007/978-3-031-42402-1_6 · Cited by 9
Geropoulos, G., Moschonas, S., Fanariotis, G., Koltsida, A., Madouros, N., Koumadoraki, E., Triantafyllidis, K. K., Kechagias, K. S., Koimtzis, G. D., Giannis, D., et al. (2023) · Medicina · doi:10.3390/medicina60010031 · Cited by 3
“The reference lists and all previously published systematic reviews were thoroughly searched for missed studies eligible for inclusion based on the “snowball” methodology [8].”
MethodSubversion of digital Darwinism in public healthcare facilities in South Africa
Shibambu, A. (2024) · Africa’s Public Service Delivery and Performance Review · doi:10.4102/apsdpr.v12i1.745 · Cited by 4
User-Like Bots for Cognitive Automation: A Survey
Gidey, H. K., Hillmann, P., Karcher, A., & Knoll, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53966-4_29 · Cited by 5
Moghaddam, M. T., & Hansen, M. M. (2023) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1145/3603166.3632549 · Cited by 5
“Bi-directional snowballing [27] from the seven take-off papers resulted in an additional eight papers, resulting in a total of 15 papers included ([15], [29], [19], [4], [9], [18], [8], [14], [12], [10], [26], [25], [13], [24], [23]).”
BackgroundMethodA catalog of quality criteria to guide the assessment of applications' privacy policies
Terra, A., Vilela, J., & Peixoto, M. (2022) · Workshop em Engenharia de Requisitos (WER) · doi:10.29327/1298262.25-21 · Cited by 3
“Claes Wohlin.”
MethodInfluentialCollado, J. R. N. (2024) · Town Planning Review · doi:10.3828/tpr.2024.17 · Cited by 3
Benefits and Challenges of the Behavior-Driven Development: A Systematic Literature Review
Arredondo-Reyes, V. M., Domínguez-Isidro, S., Sánchez-García, Á. J., et al. (2023) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft58849.2023.00016 · Cited by 3
“Finally, Snowballing was carried out in a backward and forward iteration, on the guideline described by Wohlin [7] was used to obtain further data.”
Methodde Campos, V., David, J., Ströele, V., & Braga, R. M. M. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2713 · Cited by 8
“Following the guidelines proposed by Wohlin 17 for the Snowballing procedure and the hybrid systematic literature mapping guidelines, 18,19 we employed a combination of database search and snowballing techniques.”
InfluentialLungu, S., & Nyirenda, M. (2024) · Open Journal of Applied Sciences · doi:10.4236/ojapps.2024.149167 · Cited by 16
Davis, B. J., Reid, A. D., & Rogers, B. (2024) · Global Sustainability · doi:10.1017/sus.2024.44 · Cited by 3
“Six additional publications were sourced via backwards bibliographic snowballing (see Wohlin, 2014) using the reference list in Quinn et al. (2020).”
On the Use of Visual Perception for Unmanned Surface Vehicles Navigation
Salgado, A., Bezerra, A. A., Clua, E., & Gonçalves, L. M. G. (2024) · Latin American Robotics Symposium, Brazilian Symposium on Robotics and Workshop on Robotics in Education (LARS/SBR/WRE) · doi:10.1109/lars64411.2024.10786469 · Cited by 3
“There are other approaches for determining the state of the art in some area or topic, as the snowballing [22], which is not the case here.”
Medina, M. J., Thoo, Y.-J., Baudet, C., Froehlich, J. E., Ruffieux, N., & Lalanne, D. (2025) · ACM Transactions on Accessible Computing · doi:10.1145/3726531 · Cited by 7
“…initial set of 3,378 documents ( N = 6); (2) adapting and expanding our search query to reach additional SLRs on ACM Digital Library (DL) and IEEE Xplore ( N = 5); and (3) performing a backward snowballing [204] by examining cited references of the 11 SLRs obtained in the preceding steps ( N = 1).”
Crescêncio, M., Biz, A. A., & Todesco, J. L. (2024) · Journal of Smart Tourism · doi:10.52255/smarttourism.2024.4.1.4 · Cited by 3
“This led to the adoption of a snowballing approach (Wohlin, 2014), utilizing the reference lists of papers to identify additional relevant studies.”
Educational process mining: literature classification, gaps, and emerging opportunities
Semler, R. F., Semler, J. R., Wehrmeister, M. A., Southier, L. F. P., Casanova, D., & Teixeira, M. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1543761 · Cited by 3
SoK: Security of EMV Contactless Payment Systems
Nezhad, M. M., Hao, F., Epiphaniou, G., et al. (2025) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp63326.2025.00033 · Cited by 3
“We used a reference chaining (snowballing) methodology [89], as it offers a systematic and comprehensive approach to identify key studies on contactless payment attacks by leveraging the citation networks of foundational research, involving tracing citations both forward and backward until no new…”
A Survey of Visual Analytics Research for Improving Training Data Quality
Yang, W., Chen, C., Zhu, J., Li, L., Liu, P., & Liu, S. (2023) · Journal of Computer-Aided Design & Computer Graphics · doi:10.3724/sp.j.1089.2023.2023-00321 · Cited by 10
“本文采用滚雪球的文献 调研方式 [24] , 涉及的文献范围和筛选流程如图 2 所示 .”
A Canvas for Establishing Global Software Development Collaborations
Smirnova, I. N., Münch, J., & Stupperich, M. (2014) · Communications in computer and information science · doi:10.1007/978-3-319-11958-8_7 · Cited by 5
“Based on Webster and Watson [1] as well as Wohlin [47] the starting point for the backward snowballing research approach is the analysis of main contributions to the topic.”
BackgroundMethodSystem of Systems Requirements: A Systematic Literature Review using Snowballing
de Lima, R. M., de Vargas, D., & Fontoura, L. M. (2017) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2017-114 · Cited by 3
“to identify additional studies about that subject, looking at where papers are actually referenced and cited [8].”
BackgroundMethodInfluentialScience communication scholarship in Lithuania: A scoping study
Valinčiūtė, A. (2018) · Studies in Communication Sciences · doi:10.24434/j.scoms.2017.02.002 · Cited by 3
“The second stage of sampling involved what is known as the snowballing technique, where article reference lists are checked to identify additional relevant literature (Wohlin, 2014).”
BackgroundA Preliminary Systematic Mapping Study of Human Competitiveness of SBSE
Souza, J., Araújo, A. A., Saraiva, R., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99241-9_6 · Cited by 3
Data digitalisation in the mining industry – a scoping review protocol
Duarte, J., Rodrigues, F., & Baptista, J. S. (2019) · International Journal of Occupational and Environmental Safety · doi:10.24840/2184-0954_003.001_0006 · Cited by 3
“This snowballing technique procedure will be repeated in the new selected articles until no more relevant information is obtained (Wohlin, 2014).”
MethodReview-Based User Profiling: A Systematic Mapping Study
Dong, X. L., Li, T., Li, X., et al. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-20618-5_16 · Cited by 3
Performance Testing in Mobile Application
Norberto, M., Gaedicke, L., Bernardino, M., et al. (2019) · doi:10.1145/3364641.3364653 · Cited by 3
“Ameaças ao resultado do estudo foram identificadas no SLM reali-zada, as quais foram categorizadas conforme Hyman [9] e Wohlin [28]: Validade de Construção: Pode haver uma possível exclusão de estudos relevantes.”
BackgroundNatural Language Processing for scoring open-ended questions : a systematic review
Kerkhof, R. G. (2020) · Cited by 4
Automatic support for the identification of infeasible testing requirements
Neto, J. C. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3402646 · Cited by 3
Brito, O. D. O., Aguiar, Y. P. C., & de Moura, H. P. (2020) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/cbie.sbie.2020.232 · Cited by 3
“O segundo ciclo, escopo deste artigo, ampliou a base de consulta para a literatura internacional a partir de um processo de snowball [Wohlin, 2014] e uma revisão sistemática da literatura [Cresswell, 2013], assim como aprofundou a validação da taxonomia.”
BackgroundCouto, T., Marczak, S. D. S., & Rocha, F. G. (2020) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3439961.3440000 · Cited by 4
“Os autores defendem, indepen-dente da ordem de execução, o uso combinado de uma Revisão Sistemática [15] seguida, opcionalmente, de um Snowballing [27] para consolidar o conhecimento científico, junto à uma revisão da Grey Literature , para considerar o conhecimento invisível disponível na web ,…”
BackgroundA systematic mapping study of security, trust and privacy in clouds
Odun-Ayo, I. A., Williams, T.-A., & Yahaya, J. B. (2021) · Bulletin of Electrical Engineering and Informatics · doi:10.11591/eei.v10i3.1887 · Cited by 4
“Afterwards, a backward snowballing is utilized in enhancing the search process [34].”
MethodAesthetic Flexibility : In Industrial Design Practice
Andersson, T. (2021) · Linköping studies in science and technology. Dissertations · doi:10.3384/diss.diva-173826 · Cited by 4
Business Model Flexibility and Software-intensive Companies: Opportunities and Challenges
Wilson, M., Wnuk, K., & Bengtsson, L. (2021) · e-Informatica Software Engineering Journal · doi:10.37190/e-inf210109 · Cited by 3
Let the Computer Say NO! The Neglected Potential of Policy Definition Languages for Data Sovereignty
Bartsch, J., Dehling, T., Lauf, F., Meister, S., & Sunyaev, A. (2022) · DuD-Fachbeiträge · doi:10.1007/978-3-658-33306-5_22 · Cited by 5
Software Testability (Its Benefits, Limitations, and Facilitation)
Mona, J. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-1412-6_23 · Cited by 3
A Literature Study to Characterize Continuous Experimentation in Software Engineering
Erthal, V. M., de Souza, B. P., dos Santos, P. S. M., & Travassos, G. H. (2022) · Congresso Ibero-Americano em Engenharia de Software (CIbSE) · doi:10.5753/cibse.2022.20959 · Cited by 5
“The snowballing technique searches for a research theme related to the initial articles by looking at those referenced by the initial set (backward) and those referred to (forward) [Wöhlin 2014].”
MethodOut-of-Pocket Health Care Expenditures in Uzbekistan: Progress and Reform Priorities
Cho, M. J., & Haverkort, E. (2023) · IntechOpen eBooks · doi:10.5772/intechopen.110022 · Cited by 8
Citation Gecko technology for research and entrepreneurship
Gupta, V. (2023) · Library Hi Tech News · doi:10.1108/lhtn-04-2023-0060 · Cited by 3
Toward Understanding Work Characteristics in Software Engineering
de Magalhães, C. V. C. (2017) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3011286.3011306 · Cited by 5
Privacy Salience: Taxonomies and Research Opportunities
Williams, M., Nurse, J. R. C., & Creese, S. (2016) · IFIP advances in information and communication technology · doi:10.1007/978-3-319-55783-0_18 · Cited by 4
“This was performed in the systematic method of Wohlin [53], with extensive backwards and forwards snowballing following our database search.”
Challenges in Context-Aware Requirements Modeling: A Systematic Literature Review
Li, Y., Peng, R., & Wang, B. (2018) · Communications in computer and information science · doi:10.1007/978-981-10-7796-8_11 · Cited by 3
Software Test Data Visualization with Heatmaps : an Initial Survey
Strandberg, P. E. (2017) · Cited by 3
Carvallo, J. P., Crespo-Martínez, E., Vargas, F., et al. (2018) · International Conference on Information Systems and Computer Science (INCISCOS) · doi:10.1109/inciscos.2018.00054 · Cited by 3
“[20] Wohlin, C.”
COMPETITIVE ADVANTAGE DURING INDUSTRY 4.0: THE CASE FOR SOUTH AFRICAN MANUFACTURING SMES
Futcher, M., & Sunjka, B. P. (2018) · Cited by 3
Human-Computer Interaction Factors in Software Ecosystems
dos Santos, M. A., Fernandes, J. C., Villela, M. L. B., et al. (2019) · doi:10.1145/3330204.3330247 · Cited by 3
“Para que a pesquisa ficasse mais completa, na etapa 5, foi aplicada a técnica backward snowballing [20] nos artigos que passaram pelo segundo filtro.”
BackgroundSurvey on Challenges and Achievements in Context-Aware Requirement Modeling
Li, Y., Peng, R., & Wang, B. (2020) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2019edr0002 · Cited by 3
“A method proposed in [19] was adopted to conduct the snowballing.”
MethodEnhancing Social Ties Through Manual Player Matchmaking in Online Multiplayer Games
Riyadh, M., Arya, A., Chan, G., & Imran, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60128-7_52 · Cited by 5
Internet of Things in Education for Sustainable Development
Piccolo, L. S. G., Néris, V. P. D. A., Menezes, L. M. D. S., & Néris, V. P. D. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98388-8_6 · Cited by 3
“From the selected articles in this first search, an iteration of backward snow-balling [24] was performed to identify other relevant works.”
MethodEnd-User Development Landscape: A Tour into Tailoring Software Research
Corrêa, C. M., & Silveira, M. S. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2086996 · Cited by 3
“At the bibliometric data compilation step, we performed a literature review using the snowballing procedure (Wohlin, 2014) to select the studies addressing tailoring by end-users and build our database.”
BackgroundMethodInfluentialCioffi, A., & Peroni, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16802-4_42 · Cited by 11
“It started with a pre-processing phase in which one decided which programmatic search strategy to adopt (we propose a citation-based search strategy [30]), which articles in the literature to use as seed papers for starting the search process [18], and which keywords to use for additional searching of the relevant lit…”
BackgroundCancer research and innovation: conceptualising a persistent anomaly
Xu, Y., & Gagliardi, D. (2023) · Innovation · doi:10.1080/14479338.2023.2204322 · Cited by 3
“This methodology is also used increasingly in engineering (Jalali & Wohlin, 2012; Wohlin, 2014).”
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.