Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 3,001–3,100 of 3,934
The Adoption of Learning Experience Design Tools in Classroom Planning Activity
Queiros, L. M., Bouckaert, Y. H. S. M. S., Oliveira, I. V. P. D., et al. (2019) · Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21) · doi:10.1145/3362789.3362792 · Cited by 1
“Wang Yongxing [28] 2008 Blended Learning Design for Software Engineering Course Design”
BackgroundGestão pela qualidade total: Implantação do Programa 5S na Administração Pública
de Araújo, V. S., & França, S. L. B. (2020) · Editora Poisson eBooks · doi:10.36229/978-85-7042-210-1.cap.01 · Cited by 2
A Study on Organizational IT Security in Mobile Software Ecosystems Literature
Steglich, C., Majdenbaum, A., Marczak, S. D. S., & dos Santos, R. P. (2020) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c50368.2020.00047 · Cited by 1
“To conduct the snowballing procedure, we once again followed the steps conducted in a preliminary study of ours on MSECO [8], attending the forward and backward snowballing guidelines by Wohlin [13].”
MethodHeik, D., Ghofrani, J., & Reichelt, D. (2020) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin45582.2020.9442143 · Cited by 1
“Furthermore, we are going to apply a deeper analysis of proposed results through a systematic literature review [33].”
MethodClarifying terminology for collaborative innovation and development
Santonen, T. (2021) · Cited by 1
CHINESE PRODUCT-SERVICE SYSTEM INNOVATIONS ENABLED VIA GOVERNMENTAL POLICIES: THE E-SCOOTER CASE
Yan, Z., Larsson, T. C., & Larsson, A. (2021) · Proceedings of the Design Society · doi:10.1017/pds.2021.61 · Cited by 2
“Relevant contributions were collected through a systematic search in the Scopus and Google Scholar databases and further complemented through snowballing (Wohlin, 2014) to find additional literature.”
MethodMapeamento sistemático sobre conflitos, negociação e decisão em relacionamentos interorganizacionais
Ribeiro, B. B., & Diirr, B. (2021) · doi:10.5753/ersirj.2021.16973 · Cited by 1
“Após isso, os artigos aceitos foram lidos integralmente, restando 33 artigos no MSL. Por fim, realizou-se o Backward Snowballing (Wholin, 2014), que analisa as referências de cada estudo pré-selecionado para obter mais estudos relevantes à pesquisa.”
BackgroundTaxonomia para Avaliação de Recursos Digitais de Aprendizagem - TARDA - Versão 2.0
Brito, O. D. O., Aguiar, Y. P. C., & de Moura, H. P. (2021) · RISTI - Revista Ibérica de Sistemas e Tecnologias de Informação · doi:10.17013/risti.42.120-135 · Cited by 1
“O segundo ciclo, escopo deste artigo, ampliou a base de consulta para a literatura internacional a partir de um processo de snowball (Wohlin, 2014) e uma Revisão Sistemática da Literatura (Cresswell, 2013), assim como aprofundou a validação da taxonomia.”
BackgroundHaq, M. I. (2021) · ISLAMICA Jurnal Studi Keislaman · doi:10.15642/islamica.2021.16.1.30-57 · Cited by 1
Design Automation for Additive Manufacturing : A Multi-Disciplinary Optimization Approach
Wiberg, A. (2021) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789179291082 · Cited by 3
A Meta-analytical Comparison of Energy Consumed by Two Different Programming Languages
Hamizi, I., Bakare, A., Fraz, K., Dlamini, G., & Kholmatova, Z. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-93135-3_12 · Cited by 1
Costa, D., Guedes, J. C., & Baptista, J. S. (2020) · International Journal of Occupational and Environmental Safety · doi:10.24840/2184-0954_004.002_0006 · Cited by 1
Azemour, S., Ayache, M., Bakkali, H. E., et al. (2022) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-97652-1_23 · Cited by 1
A Systematic Review About Requirements Elicitation for Multi-Agent Systems
Gerstberger, W. S., Filho, I. P. D. S., & Guedes, G. T. A. (2022) · Smart innovation, systems and technologies · doi:10.1007/978-981-19-3359-2_22 · Cited by 1
Using goal–question–metric to compare research and practice perspectives on regression testing
Minhas, N. M., Koppula, T. R., Petersen, K., & Börstler, J. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2506 · Cited by 2
Batista, G. O. D. S., Monteiro, M. D. S., & Salgado, L. C. D. C. (2022) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3554364.3559138 · Cited by 4
“Sendo, posterior a seleção, a aplicação de uma técnica chamada forward snowballing [53].”
A Provenance-based Execution Strategy for Variant GPU-accelerated Scientific Workflows in Clouds
Stockinger, M. B., Guerine, M. A., de Paula, U., Santiago, F. T., Frota, Y. A. R., Rosseti, I., Plastino, A., & de Oliveira, D. (2022) · Journal of Grid Computing · doi:10.1007/s10723-022-09625-y · Cited by 2
“For diversity reasons, it is recommended [75] that the articles in the seed set, which will have the references analyzed, should belong to different research groups, dates or communities.”
BackgroundMethodTennant, R., Graham, J., Mercer, K., Ansermino, J. M., & Burns, C. M. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-065429 · Cited by 2
Madrigar, T. T., Colanzi, T. E., OIZUMI, W. N., Okada, L. F., & Garcia, A. (2022) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2021.1903 · Cited by 1
“[40]: internal, external, construct and conclusion.”
Weshahi, S. J. S. A. (2022) · Global Journal of Human Resource Management · doi:10.37745/gjhrm.2013/vol10n61522 · Cited by 1
Uma análise sistemática sobre o uso de chatbots para ensino de computação no Brasil
UFGD, Melo, S. M., Pessoa, E. B., & Paschoal, L. N. (2003) · RENOTE · doi:10.22456/1679-1916.129155 · Cited by 5
Tóth, B., & Oldal, L. G. (2022) · IEEE International Symposium on Intelligent Systems and Informatics (SISY) · doi:10.1109/sisy56759.2022.10036252 · Cited by 1
“…in the index period have been found, two methods have been combined: expand the database, and indirectly the set of search terms, using the articles already found [21] • a text analysis method to find more precise search terms through our ever-expanding whether they meet the inclusion criteria.”
MethodMapping Public-Planner Conflicts in SUDS Implementation Using Cultural Dimensions—A Case Study
Thodesen, B., Andenæs, E., Bohne, R. A., & Kvande, T. (2023) · Urban Science · doi:10.3390/urbansci7020061 · Cited by 3
“Using the “snowballing” method [49], the bibliographies of retrieved articles were also consulted to find other relevant studies.”
MethodBlockchain Governance Design a Computer Science Perspective
Wang, Y., Kaya, F., & Gordijn, J. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-34985-0_18 · Cited by 2
Um Estudo sobre Colaboração em Ecossistemas Digitais de Inovação Social
Pinheiro, M. C., Chueri, L., & dos Santos, R. P. (2022) · iSys - Brazilian Journal of Information Systems · doi:10.5753/isys.2022.2364 · Cited by 1
Systematic literature review of the black friday promotional strategy
Guerra, A. A. C., & Fouto, N. M. M. D. (2023) · Retail Management Review · doi:10.53946/rmr.v4i1.26 · Cited by 2
Teixeira, P. G., Lebtag, B. G. A., dos Santos, R. P., et al. (2023) · Annual System of Systems Engineering Conference (SoSE) · doi:10.1109/sose59841.2023.10178558 · Cited by 1
“The search strategy combined, at first, automatic search and backward snowballing in the selected studies [8].”
MethodOn the prediction of source code design problems: A systematic mapping study
Silva, R. K., Farias, K., & Kunst, R. (2023) · Journal of Applied Research and Technology · doi:10.22201/icat.24486736e.2023.21.3.1749 · Cited by 1
“To mitigate this threat, studies have been added using the snowballing method (both backward and forward) (Jalali & Wohlin et al., 2012; Wohlin, 2014).”
MethodAn Overview of the Understanding and Use of Assessment Methods in Benefits Management
Patón-Romero, J. D., & Hannay, J. E. (2023) · IEEE Conference on Business Informatics · doi:10.1109/cbi58679.2023.10187441 · Cited by 1
“, the snowballing approach [41] will be applied.”
MethodGender and Styles of Leadership in Higher Education
Renigunta, S., & Mohammed, L. A. (2022) · International journal of Emerging Issues in Social Science Arts and Humanities · doi:10.60072/ijeissah.2022.v1i01.004 · Cited by 1
Loma, E. A. Q., & Perdomo, V. P. T. (2023) · Ciencia Latina Revista Científica Multidisciplinar · doi:10.37811/cl_rcm.v7i4.7558 · Cited by 1
Measuring Circularity: The Gordian Knot of the 21st Century
Wierikx, A., Ravesteijn, P., Raassens, N., & Alblas, A. A. (2023) · Communications of the IIMA · doi:10.58729/1941-6687.1439 · Cited by 2
Neurological semiology learning and technological-digital perspectives: a scoping review
Berton, G. G., Dagnese, L. B., Guerra, G. C., Volino, G. C., HENICKA, J. C. P., Rodrigues, L. L. R., Castel, G. M. D., Saltiél, R. M. F., Giovelli, M. E., & Zanatta, A. L. (2023) · Journal of Health Informatics · doi:10.59681/2175-4411.v15.i2.2023.995 · Cited by 2
Using a hybrid methodology for literature review: a case study in depression research
Abdallah, S., & Khalil, A. (2023) · Information Discovery and Delivery · doi:10.1108/idd-03-2022-0020 · Cited by 4
Tierra-Arévalo, J. M., Pardo-Ferreira, M. D. C., Herrera-Pérez, V., et al. (2023) · Studies in systems, decision and control · doi:10.1007/978-3-031-38277-2_65 · Cited by 1
Work Accidents Related to Heavy Equipment in the Open Pit Extractive Industry: A Systematic Review
Souza, J., Branco, J. C., & Baptista, J. S. (2023) · Studies in systems, decision and control · doi:10.1007/978-3-031-38277-2_62 · Cited by 1
Maepa, M. R., & Moeti, M. N. (2024) · IntechOpen eBooks · doi:10.5772/intechopen.113329 · Cited by 3
Aşan, C. (2023) · Mersin University Journal of Maritime Faculty · doi:10.47512/meujmaf.1370274 · Cited by 4
Raffaghelli, J. E., & Crudele, F. (2023) · Media Education · doi:10.36253/me-13801 · Cited by 8
Blockchain-based Supply Chain Management Systems: A Systematic Mapping Study of Academic Research
Haughton, O., Campbell, C. E.-A., Walcott, T. H., & Neaga, I. (2023) · International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA) · doi:10.1109/contesa61248.2023.10384965 · Cited by 1
“The initial search results were then filtered based on inclusion/exclusion criteria, detailed subsequently, and subjected to a rigorous snowballing process as described by Wohlin [15], employing both forward and backward snowballing until no further relevant publications were identified.”
MethodTowards Design Guidelines for IoT Applications considering Elderly Users
Rodrigues, S. S., Fortes, R. P. D. M., & Rodrigues, K. R. D. H. (2023) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3638067.3638139 · Cited by 2
Sanchís-Soler, G., Tortosa‐Martínez, J., Sebastiá-Amat, S., Chulvi‐Medrano, I., & Cortell-Tormo, J. M. (2024) · Healthcare · doi:10.3390/healthcare12030397 · Cited by 4
“Additionally, a snowballing technique will be used (the reference list of an article or the citations of the article can be used to identify additional articles) [28].”
MethodCurriculum Design Principles for Equity in Engineering (Fundamental, Diversity)
Cunningham, C. M., & Kelly, G. J. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--40576 · Cited by 2
Rivas, L. Y. C., Domínguez, E. L., Velázquez, Y. H., et al. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-50590-4_21 · Cited by 1
Adil, M., Fronza, I., & Pahl, C. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194024500098 · Cited by 1
Kousar, A., Khan, S. U. R., Hussain, S., Rahim, M. A. B. U., Wang, W., & Ibrahim, N. (2023) · International Multi-Topic Conference · doi:10.1109/inmic60434.2023.10465949 · Cited by 2
“Note that snowballing is a systematic way that aims to find the most relevant studies from the references of the primary studies [13].”
BackgroundMethodOPTIMALISASI E-GOVERNMENT DALAM PEMBERANTASAN KORUPSI
Syihabuddin, M. A., Nugroho, R., Fitriana, A. R. D., & Ilahiyyah, I. (2024) · Jurnal Kebijakan Publik · doi:10.31258/jkp.v15i1.8416 · Cited by 7
“Tahap kedua melakukan pencarian dan pe-nyaringan literatur yang sesuai dengan topik pen-elitian dengan metode bola salju dengan prosedur (Wohlin, 2014) Tahap proses selanjutnya melibtakan penca-rian makalah dari penerbit seperi, Springer, Taylor and Francis, Emerald Insight, Wiley-Blackwell, dan…”
BackgroundResources and Capabilities of Malaysian Retailers’: A Systematic Review Toward Competitive Advantage
Hairani, N., & Aziz, Z. A. (2024) · The European Proceedings of Social & Behavioural Sciences · doi:10.15405/epsbs.2024.05.40 · Cited by 2
“Nonetheless, several complementary methods can be considered in search efforts (Tsafnat et al., 2014; Wohlin, 2014).”
Veronez, F. A., Celestino, J. E. M., Almeida, M. R. R. E., et al. (2024) · Revista Brasileira de Geografia Física · doi:10.26848/rbgf.v17.3.p1697-1719 · Cited by 1
“Como o universo de trabalhos identificados era muito pequeno, optou-se por realizar uma nova seleção de trabalhos utilizando o método de bola de neve, a partir das referências citadas nos trabalhos identificados (Wohlin, 2014).”
BackgroundSustainability criteria for introducing new technologies in low-income contexts
Mallalieu, A., Jonasson, A., Petersson, S., et al. (2024) · Proceedings of the Design Society · doi:10.1017/pds.2024.138 · Cited by 1
“This study was of both snowballing (Wohlin, 2014) and systematic characteristics.”
BackgroundExploring the Role of Bots in Software Development
Moguel-Sánchez, R., Palacios, C. S. M., Ocharán-Hernández, J. O., Limón, X., & Sánchez-García, Á. J. (2024) · Proceedings of the Institute for System Programming of RAS · doi:10.15514/ispras-2024-36(1)-13 · Cited by 2
Supporting Developers' Emotional Awareness: from Self-reported Emotions to Biometrics
Grassi, D. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661209 · Cited by 1
“Starting from a subset of relevant papers, [8, 12, 18, 19, 31, 33, 38, 39] I have applied snowballing, as proposed by Wohlin [40], to identify further relevant studies that I could have missed in the previous stage.”
MethodDeveloping a Hybrid Deployment Model for Highly Available Manufacturing Execution Systems
Furmanek, L., Lins, S., Blume, M., & Sunyaev, A. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569530 · Cited by 2
“A snowballing based literature review [17] was selected to lay the foundation for the protection objectives for highly available hybrid MES.”
BackgroundMethodInfluentialDefinitions and Computational Properties of OCL: A Systematic Review
Rajić, G., & Sruk, V. (2024) · IEEE Access · doi:10.1109/access.2024.3428865 · Cited by 2
“When the preliminary selection set is relevant enough, the search process can be refined by forward and backward snowballing [34].”
BackgroundGamification for Manufacturing (GfM) Towards Era Industry 5.0
Baroroh, D. K., Santoso, H. B., & Van, N. T. H. (2024) · International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON) · doi:10.1109/ecti-con60892.2024.10594851 · Cited by 1
“Within the job-shop manufacturing environment, gamification enables the introduction of entertainment [24], alleviates monotony [25], and enhances engagement [26], particularly in tasks characterized by repetitive and time-consuming production processes.”
BackgroundVishnupriya, V., Meldrum, J., Kahandawa, R., Domingo, N., Shahzad, W. M., & Shen, X. (2024) · Future Cities and Environment · doi:10.5334/fce.266 · Cited by 2
Electric Vehicle Fast-Charging Software: Architectural Considerations Towards Trustworthiness
Dini, V., Tamburri, D. A., & Nitto, E. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70797-1_19 · Cited by 1
A Comprehensive Exploration of the Use of Software Testing Tools for IoT Systems
Herrera, C. G., Ferreira, V. G., Souza, S. R. S., et al. (2024) · doi:10.5753/wtf.2024.3247 · Cited by 1
Wnuk, K., Madeyski, L., Abdeen, W., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70819-0_3 · Cited by 1
Tamper-Proof Blockchain-Based Contracts for the Carriage of Goods by Road
Carreira, F., da Cunha, P. R., Barata, J., & Estima, J. (2024) · Proceedings of the International Conference on Information Systems Development · doi:10.62036/isd.2024.105 · Cited by 2
Common Characteristics Between Frailty and Myotonic Dystrophy Type 1: A Narrative Review
Garmendia, J., Labayru, G., Barreto, P. D. S., Vergara, I., de Munaín, A. L., & Sistiaga, A. (2024) · Aging and Disease · doi:10.14336/ad.2024.0950 · Cited by 2
“To further expand the range of studies, a forward and backward snowballing approach was employed [33].”
Palavani, L. B., Verly, G., Borges, P., et al. (2024) · Neurosurgical Review · doi:10.1007/s10143-024-02839-6 · Cited by 1
“We also manually searched the references of the retrieved studies, previous systematic reviews, and meta-analyses for any additional relevant studies [48].”
Security Mechanisms Used in Systems Based on Zero Trust Architecture: A Systematic Mapping
Manzano, C., Márquez, G., & Hernán Astudillo, R. (2024) · Latin American Computer Conference (CLEI) · doi:10.1109/clei64178.2024.10700484 · Cited by 2
Addressing the Synchronization Challenge in Cypress End-to-End Tests
de Moura, T. S., Alves, E. L. G., Felipe, R. L. S., et al. (2024) · doi:10.5753/sbes.2024.3298 · Cited by 1
“To systematically identify relevant waiting mechanisms, we conducted a literature mapping using common keyword searches combined with snowballing, a technique in which references from selected articles are reviewed to identify additional relevant studies, ensuring a comprehensive review [40].”
MethodAgent-Based Modeling and Simulation for Collaborative Software: A Systematic Literature Review
Sandria-Flores, D., Mezura-Godoy, C., & Benítez–Guerrero, E. (2024) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft63288.2024.00025 · Cited by 1
Halker, N. S. M., Brandão, A. L., & Gois, J. P. (2024) · doi:10.5753/sbgames.2024.240528 · Cited by 1
A Comprehensive Review of User Interaction for Recommendation Systems
Martins, C. A., Dorneles, C. F., & Winckler, M. A. A. (2024) · iSys - Brazilian Journal of Information Systems · doi:10.5753/isys.2024.4064 · Cited by 1
“A snowballing process [Wohlin 2014] (consisting of extracting studies related to a start set of papers, using a backward/forward strategy, taking the reference list to identify new works to include, and looking forward to identifying new works based on those papers citing the paper being examined)…”
Nazário, M., Bonifácio, R., de Souza, C. R. B., Kamei, F., & Pinto, G. (2025) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2025.4378 · Cited by 1
Meade, A., Santero, M., Savall-Esteve, O., Bracchiglione, J., Leache, L., Selva, A., Macias, I., Cerdà, P., & Bonfill Cosp, X. (2025) · Journal of Gastrointestinal Cancer · doi:10.1007/s12029-024-01155-y · Cited by 1
“Furthermore, we manually reviewed the reference lists of relevant studies to identify any additional relevant articles [35].”
Front-End Loading, an indicator of project management success?
Bakke, C., & Johansen, A. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.02.312 · Cited by 1
Stavrou, N., Memos, N. A., Filippatos, C., Sergentanis, T. Ν., Zagouri, F., Gavriatopoulou, M., & Ntanasis‐Stathopoulos, I. (2025) · Journal of Gastrointestinal Cancer · doi:10.1007/s12029-025-01210-2 · Cited by 1
“Additionally, a comprehensive and systematic snowball approach was used to capture all relevant records by reviewing the reference lists of included studies, ensuring comprehensive coverage and minimizing the risk of omitting previously cited literature [21].”
Using large Language models as a road map for establishing core values in a legal vacuum
Theunissen, T., & Duijvestijn, L. (2025) · AI and Ethics · doi:10.1007/s43681-025-00706-8 · Cited by 1
“To gather data on 500 non-profits, we employed a forward snowballing technique [ 27 ], starting from various existing lists and expanding our data set.”
Assessing Energy Flexibility: A Comparative Review of Nigeria and the United Kingdom
Akporhonor, G. K., & Mirzania, P. (2025) · Springer eBooks · doi:10.1007/978-3-031-83165-2_4 · Cited by 1
Schulte, J., Bäckman, E., Giunta, V., & Salirwe, M. (2025) · Business Strategy and the Environment · doi:10.1002/bse.4310 · Cited by 9
“From those, backward snowballing was done to also cover central articles published in other journals (Wohlin 2014).”
Unveiling legal complexity: a systematic review on the visual analytics of legal corpora
Mentzingen, H., António, N., & Bação, F. (2025) · International Review of Law Computers & Technology · doi:10.1080/13600869.2025.2497630 · Cited by 3
“This process is known as ‘snowballing’ (Greenhalgh and Peacock 2005; Wohlin 2014) and was performed in forward and backward directions.”
Peng, X., Khan, A., Su, Y., Bai, Y., Tang, Q., & Li, J. (2025) · Journal of Business Ethics · doi:10.1007/s10551-025-06013-w · Cited by 5
Gender Differences in Planning Poker: A Preliminary Approach with Epistemic Network Analysis
Palma-Ramírez, D., Miranda, D., Noël, R., Ochôa, X., Cechinel, C., & Muñoz, R. (2025) · Lecture notes in educational technology · doi:10.1007/978-981-96-3698-3_15 · Cited by 1
Pardo, C., Pérez, E. N., & Brieva, E. D. C. S. (2025) · Applied Sciences · doi:10.3390/app15105533 · Cited by 1
A Survey on the Development of Human-Centered Artificial Agents for Fighting Games
Tavares, M., de Carvalho, J. M., & Pinho, A. J. (2025) · IEEE Transactions on Games · doi:10.1109/tg.2025.3577552 · Cited by 1
“This is akin to the backward snowballing approach for systematic literature reviews [48].”
Ensuring Syntactic Interoperability Using Consumer‐Driven Contract Testing
Schwarz, G., Quast, F., & Riehle, D. (2025) · Software Testing Verification and Reliability · doi:10.1002/stvr.70006 · Cited by 9
Privacy Preserving Identity Federation: A Literature Study
Bumiller, A., Kozyri, E., & Dagenborg, H. (2026) · ACM Computing Surveys · doi:10.1145/3745018 · Cited by 8
“This hybrid search strategy combining structured databases and grey literature sources with rigorous filtering is aligned with best practices in systematic literature reviews [53, 116].”
Strelnikova, E. V., Popov, V. A., Горбатова, Л. Н., Gorbatova, M. A., & Dubinina, A. (2025) · Parodontologiya · doi:10.33925/1683-3759-2025-1050 · Cited by 2
Assessing the relationship between dietary factors and hair health: A systematic review
Gomes, N., Silva, N., & Teixeira, B. (2025) · Nutrition and Health · doi:10.1177/02601060251367206
“…fi nalizing the inclusion and exclusion criteria, a comprehensive analysis of the selected references was conducted using the snowballing technique (Wohlin, 2014) to identify additional relevant studies The number of studies excluded and the reasons for this are presented in the PRISMA fl…”
Celebrating new life – A literature review of research on birth rites in Western societies
Høeg, I. M., & Jarnkvist, K. (2025) · International Journal of Sociology and Social Policy · doi:10.1108/ijssp-09-2024-0449 · Cited by 2
Samothraki, A. L., Evangelinos, K. I., & Nikolaou, I. E. (2025) · Human Rights Review · doi:10.1007/s12142-025-00749-8 · Cited by 2
“In order to reliably collect relevant literature and avoid a biased approach, both backward and forward citation analysis is used to identify critical and influential studies in the literature (Wohlin, 2014 ).”
The Temporal Trajectory and Spatial Variability of Conservation and Management of Primates in India
Anand, S. (2025) · International Journal of Primatology · doi:10.1007/s10764-025-00517-y · Cited by 1
“Following identification of seed articles, backward snow-balling was conducted by systematically reviewing reference lists to identify additional relevant publications (Greenhalgh & Peacock, 2005; Wohlin, 2014).”
A Set of Artifacts and Models to Support Requirements Communication Based on Perspectives
Oran, A. C. (2017) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3011286.3011303 · Cited by 1
“We applied the Forward Snowballing technique to search for papers citing our initial list of studies as a reference and Backward Snowballing to search for new papers in our list of references [25].”
BackgroundMethodAgile Testing in Brazil: A Systematic Mapping
Farias, J. L., de Vasconcelos, A. M. L., & Junior, I. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-73673-0_3 · Cited by 1
Revisões sistemáticas da literatura: parte I
Neves, L. D. O., Bernardini, S. P., Ruschel, R. C., & Moreira, D. D. C. (2018) · PARC Pesquisa em Arquitetura e Construção · doi:10.20396/parc.v8i3.8651561 · Cited by 2
Interpersonal Trust in Virtual Software Development Teams: A Systematic Mapping Study
Zapata, S., Barros‐Justo, J. L., Romagnano, M. R., & Arrón, M. (2017) · El Servicio de Difusión de la Creación Intelectual (National University of La Plata) · Cited by 1
Wilson, M., & Wnuk, K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02131-3_26 · Cited by 2
“The Systematic Literature Review (SLR) methodology is based on the guidelines for snowballing literature search proposed by Wohlin [27], outlined below 3 .”
Squat and Patellofemoral Pain: protocol for a systematic review
Pereira, P. M., Duarte, J., Ferraz, J., Costa, J. T., & da Conceição, F. A. (2019) · International Journal of Occupational and Environmental Safety · doi:10.24840/2184-0954_003.002_0001 · Cited by 1
“In the succeeding steps, articles, and other works published before 2014 will be considered, and snowballing procedures (Wohlin & Claes, 2014) will be followed.”
BackgroundDavis, K., & Davidson, J. (2019) · Palliative Medicine and Hospice Care - Open Journal · doi:10.17140/pmhcoj-5-131 · Cited by 1
Combinatorial Testing of Web Services
Rehman, B., Alam, K. A., Rehman, M., et al. (2019) · International Conference on Future Networks and Distributed Systems (ICFNDS) · doi:10.1145/3341325.3342009 · Cited by 1
“This phase implies by using reference citations or a list of identified research papers to extra searching other sources [13].”
BackgroundRevisão Sistemática de Literatura Sobre Métricas de Sustentabilidade
Melchio, C., & Henriqson, É. (2020) · Revista FSA · doi:10.12819/2020.17.2.2 · Cited by 1
“Na terceira e última etapa buscaram-se os artigos referenciados pelos estudos encontrados nas buscas para maximizar a abrangência do estudo, realizando uma única iteração da técnica de snowballing (WOHLIN, 2014).”
BackgroundUma Proposta de Boas Práticas para o Ensino Colaborativo de Programação de Computadores
Zanin, A., Sparremberger, A. S., & Barbosa, J. L. V. (2019) · RENOTE · doi:10.22456/1679-1916.99292 · Cited by 1
Pesquisando sobre Transparência: Resultados Preliminares de um Mapeamento Sistemático
Suescún, E., & Leite, J. C. S. D. P. (2017) · doi:10.5753/wtrans.2017.3120 · Cited by 1
“Acreditamos que os artigos achados servem como ponto de partida para realizar a busca manual sugerida por Zhang et al. (2011), o piloto também segue as sugestões propostas por Wohlin (2014), que propõe o uso do Google Scholar, por exemplo, para identificar artigos base.”
BackgroundSoares, E. C., Viveiro, D. A., Mourão, S. K. R., Sousa, M. D. O., Barbosa, I. D. S., & Soares, D. R. L. (2020) · Editora Poisson eBooks · doi:10.36229/978-85-7042-210-1.cap.10 · Cited by 1
A gestão comunitária dos sistemas de abastecimento de água: Em busca dos compromissos da agenda 2030
de Souza, A. C. D. S. V., Alves, L. M. C., Rocha, B. B., Kazadi, J. B., Gonçalves, R. P., & Machado, A. V. M. (2020) · Editora Poisson eBooks · doi:10.36229/978-85-7042-210-1.cap.13 · Cited by 1
Schefer, R. P., & Furnival, A. C. (2020) · P2P E INOVAÇÃO · doi:10.21721/p2p.2020v6n2.p102-116 · Cited by 1
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.