Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 2,801–2,900 of 3,934
Trustworthiness-Related Risks in Autonomous Cyber-Physical Production Systems - A Survey
Zahid, M., Bucaioni, A., & Flammini, F. (2023) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr57506.2023.10224955 · Cited by 2
“Furthermore, to avoid missing any related articles a backward and forward snowballing was conducted [19].”
BackgroundMethodObject-Based Access: Enhancing Accessibility with Data-Driven Media
Nevsky, A. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3573381.3596500 · Cited by 3
España, S., van der Maaten, C., Gulden, J., & Pastor, Ó. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47262-6_2 · Cited by 2
Business Innovation Towards a Circular Economy
Konietzko, J. (2020) · Architecture and the Built Environment · doi:10.59490/abe.2020.22.5470 · Cited by 12
Teixeira, T. S. R., Silveira, F. F., & Guerra, E. (2023) · Computers · doi:10.3390/computers12110230 · Cited by 3
The Two Faces of AI in Green Mobile Computing: A Literature Review
Siemers, W., Sallou, J., & Cruz, L. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00053 · Cited by 3
“Following the guidelines presented by Wohlin [8], we conduct a single iteration of snowballing, encompassing both backward and forward passes for the papers in the initial set.”
BackgroundMethodInfluentialMartinez, J., Eguia, A., Urretavizcaya, I., Amparan, E., & Negro, P. (2023) · International Conference on System Reliability and Safety (ICSRS) · doi:10.1109/icsrs59833.2023.10381456 · Cited by 6
“We conducted an snowballing approach [15] where we checked referenced papers of the primary sources to identify studies that also matched our inclusion and exclusion criteria.”
MethodSilva, F. (2024) · Bellaterra Journal of Teaching & Learning Language & Literature · doi:10.5565/rev/jtl3.1294 · Cited by 2
“Having established our starting point, we employed a backward snowball-ing technique (Wohlin, 2014), utilizing reference lists to uncover additional relevant documents.”
MethodAmbiguity Resolution and Legal Compliance of Requirements: an Exploratory Study in the Literature
Netto, D., & Silva, C. (2023) · Workshop em Engenharia de Requisitos (WER) · doi:10.29327/1298356.26-17 · Cited by 2
Ensino e Aprendizagem de Programação na Educação 4.0: Um Mapeamento Sistemático da Literatura
Oliveira, K. K. D. S., Marcolino, A. D. S., Falcão, T. P., & Barbosa, E. F. (2024) · doi:10.5753/educomp.2024.237363 · Cited by 9
“…acordo com as orientações de Kitchenham e Charters [14], combinando a pesquisa automatizada utilizando strings de pesquisa em bibliotecas digitais com o método de Snow-balling [38], analisando estudos que citam o estudo já selecionado ( forward snowballing ) por meio da plataforma Google Scholar .”
BackgroundSustainable governance of natural gas exploration: A coastal forest ecosystem perspective
Bungwa, Z. S., Yanda, P. Z., & Lyimo, J. G. (2024) · Business Strategy & Development · doi:10.1002/bsd2.411 · Cited by 2
Understanding How Power Distance Affects Agile Organizations
Mattos, C. S., Dutra, E., Canedo, E. D., & Santos, G. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-70245-7_22 · Cited by 3
Guidelines for Designing IoT Applications for Older Adults
Rodrigues, S. S., Fortes, R. P. D. M., & Rodrigues, K. R. D. H. (2024) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3702038.3702082 · Cited by 4
“This section details the refinement of the guidelines, which led to an updated version of IoT_DGE (version 2.1), based on a literature review utilizing the snowballing method [65].”
InfluentialTowards an ontology of dependability Automatic search approach with Systematic Mapping
Nadia, Z., Afifi, N., & Hilal, I. (2017) · International Conference on Big Data, Cloud and Applications (BDCA) · doi:10.1145/3090354.3090379 · Cited by 2
“Among the existing guidelines, we have: (i) string research in databases [8], and (ii) documentary research using articles references or so-called "Snowballing" [15].”
MethodReconciling the Academic and Enterprise Perspectives of Design Thinking
Camposano, J. C. (2018) · Lecture notes in business information processing · doi:10.1007/978-3-319-94214-8_2 · Cited by 4
“Following the guidelines by Wohlin [31], the abstract of the papers was read first and then any other parts until a definitive decision could be taken to either include or exclude the paper.”
MethodBeyond BIM: The Evolution of an IT for Construction Course
Sands, K. S., Fiori, C. M., & Suh, M. J. (2018) · ASCE Construction Research Congress (CRC) · doi:10.1061/9780784481301.006 · Cited by 2
Adaptation of Modern Agile Practices in Global Software Engineering
Mansoor, M., Khan, M. W., Rizvi, S. S. H., Hashmani, M. A., & Zubair, M. (2019) · Advances in systems analysis, software engineering, and high performance computing book series · doi:10.4018/978-1-5225-9448-2.ch007 · Cited by 2
“This has created a rationale to explore and adopt agile development methodology in GSE.”
BackgroundInfluentialBest Value Procurement From a Contractor Point of View
Lesjø, E. S., Wondimu, P. A., & Lædre, O. (2019) · Annual Conference of the International Group for Lean Construction · doi:10.24928/2019/0190 · Cited by 2
“In order not to miss important literature, both forward and regular snowballing were used (Wohlin 2014).”
MethodEmpirical study of customer communication problem in agile requirements engineering
Gizzatullina, I. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3342511 · Cited by 2
“Systematic literature review (SLR) using database search and back-ward/forward snowballing techniques [22] was performed to ap-praise relevant studies, and to acquire evidence for investigation of customer communication problems in agile RE activities.”
Applied Data Science in Financial Industry: Natural Language Processing Techniques for Bank Policies
Spruit, M. R., & Ferati, D. (2019) · Springer proceedings in complexity · doi:10.1007/978-3-030-30809-4_32 · Cited by 2
Systematic mapping study of economic and business models of cloud services
Odun-Ayo, I. A., Williams, T.-A., Abayomi‐Alli, O., & Yahaya, J. B. (2020) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v18.i2.pp987-994 · Cited by 3
“Thereafter, a backward snowballing was carried out to complement the search process [27].”
MethodUdvikling af modelleringskompetence i læreruddannelsen
Andersen, P. U., Brandt, H., Krogh, L. B., et al. (2020) · MONA - Matematik- og Naturfagsdidaktik · doi:10.7146/mona.v2020i2.120785 · Cited by 2
Exploring the Meaning of “Usable Security”
Lennartsson, M., Kävrestad, J., & Nohlberg, M. (2020) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-57404-8_19 · Cited by 6
“Backward snowballing, as described by [82], was employed and resulted in another 21 papers, resulting in 70 papers that were included for the study.”
MethodRheumatic Diseases in Aged Homeless in US and Canada
Hicks, A. (2020) · Archives of Rheumatology & Arthritis Research · doi:10.33552/arar.2020.01.000510 · Cited by 2
A Systematic Literature Review on Model-driven Engineering for Cyber-Physical Systems
Mohamed, M. A., Kardaş, G., & Challenger, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.08644 · Cited by 8
“According to [150], forward snowballing process is realized by identifying other publications citing any of the primary studies.”
BackgroundChallenges and Opportunities on Using Games to Support IoT Systems Teaching
de Souza, B. P., & Werner, C. M. L. (2021) · Latin American Conference on Learning Technologies (LACLO) · doi:10.1109/laclo54177.2021.00093 · Cited by 2
“Snowballing é uma técnica de busca de estudos sistemáticos da literatura, em que sua aplicação se dá por meio de uma busca na lista de referências de obras ( backward ) ou citações em máquinas de buscas ( forward ) [15].”
BackgroundTesting Reading Tactics for Automated Reading Assistance: Is it Useful to Apply Old Knowledge?Preprint
Yu, Z., & Menzies, T. (2017) · arXiv (Cornell University) · Cited by 2
What we know about software testability: a survey.Preprint
Garousi, V., Felderer, M., & Kılıçaslan, F. N. (2018) · arXiv (Cornell University) · Cited by 2
An account of silence in diagnostic radiography : a cultural quilt
O'Regan, T. (2019) · University of Salford Institutional Repository (University of Salford) · Cited by 3
Safety-Critical Systems and Agile Development: A Mapping StudyPreprint
Kasauli, R., Knauss, E., Kanagwa, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07800 · Cited by 2
“Then, the first two authors went through the full texts of these remaining 69 papers collaboratively and selected 33 papers based on the inclusion and exclusion criteria in Table I, to which we added one more paper identified through snowballing [14] for consideration (indicated with +1 in Table I), resulting in a tot…”
BackgroundVisualization in the preprocessing phase: an interview study with enterprise professionalsPreprint
Milani, A. M. P., Paulovich, F. V., & Manssour, I. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07894 · Cited by 2
Landscaping Systematic Mapping Studies in Software Engineering: A Tertiary StudyPreprint
Khan, M. U., Sherin, S., Iqbal, M. Z., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10751 · Cited by 2
“to conduct the process of snowballing [32].”
Sepúlveda, S., Diéguez, M., Farías, G., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09489 · Cited by 2
“- Step 2: snowballing process (backward & forward), starting from the final list of selected papers to complement the automatic search (Wohlin 2014).”
MethodSociotechnical Aspects of Automated Recommendations: Algorithms, Ethics, and Evaluation
Paraschakis, D. (2020) · doi:10.24834/isbn.9789178770755 · Cited by 3
Does the Mark-to-Model Fair Value Measure Make Assets Impairment Noisy?Preprint
Dudycz, T., & Praźników, J. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3203045 · Cited by 2
Games to Collaboratively Explore Environmental Complexity: Designing the Virtual River Game
den Haan, R. (2020) · doi:10.3990/1.9789036549653 · Cited by 5
Patnaik, N., Dwyer, A. C., Hallett, J., & Rashid, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.02031 · Cited by 2
“To identify relevant work, we used a snowball method [69].”
MethodQualities of Quality: A Tertiary Review of Software Quality Measurement ResearchPreprint
Champion, K., Khatri, S., & Hill, B. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.13687 · Cited by 3
A Systematic Assessment on Android Third-party Library Detection ToolsPreprint
Zhan, X., Liu, T., Liu, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01964 · Cited by 2
“To investigate existing TPL detection techniques, we first follow a well-defined Systematic Literature Review (SLR) methodology [58], [59] to find related research in this area.”
MethodSocial Media: The Real Impact on Food Waste Reduction Beyond the Swipe or the Click
Hou, J. Z., Lotz, A. D., Hearn, G., & Lewis, K. (2022) · Queensland University of Technology · doi:10.5204/rep.eprints.228653 · Cited by 3
“…sources Databasedriven search (automation) • Proquest • Ebscohost • Emerald Insight • Scopus • Web of Science • ScienceDirect Backward & Forward Snowballing (manual) Following Wohlin (2014) (see Figure 1), the database-driven search approach is combined with the snowballing approach…”
MethodInfluentialInteractive Online Machine Learning
Tegen, A. (2022) · doi:10.24834/isbn.9789178772810 · Cited by 3
DepProMod: Modelo de Proceso de Despliegue de Sistemas de Software
Panizzi, M. D. (2022) · El Servicio de Difusión de la Creación Intelectual (National University of La Plata) · doi:10.35537/10915/139509 · Cited by 6
Riyadh, M. (2023) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2023-15505 · Cited by 2
Khan, A., Ali, A., Khan, J., Ullah, F., & Khan, M. A. (2023) · Research Square · doi:10.21203/rs.3.rs-3209115/v1 · Cited by 4
BIM Best Practices for Construction Health and Safety: A Systematic ReviewPreprint
Sidani, A., Martins, J. P., & Soeiro, A. A. V. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4595455 · Cited by 3
Fifteen years of extensive and inadvertent elasmobranch trade in Brazil detected by DNA toolsPreprint
Alvarenga, M. M., Bunholi, I. V., de Brito, G. R., et al. (2023) · doi:10.32942/x2n31x · Cited by 2
Diretrizes de design para aplicações de Internet das Coisas com foco em pessoas idosas
Rodrigues, S. S. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2023.tde-19122023-172515 · Cited by 3
Systematic Literature Review on Forecasting and Prediction of Technical Debt EvolutionPreprint
Ajibode, A., Yvon, A., & Temitope, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4697031 · Cited by 7
Gunawardhana, K., Kumara, B., Rathnayake, K. A. C. J. J., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-3869354/v1 · Cited by 2
A systematic review of the biomarkers associated with cognition and mood state in bipolar disorderPreprint
Pérez‐Ramos, A., Cristina, R.-L.-A., Hidalgo‐Figueroa, M., Berrocoso, E., & Pérez‐Revuelta, J. I. (2024) · Research Square · doi:10.21203/rs.3.rs-4020734/v1 · Cited by 2
Human Factors in Security Risk Analysis of Software Systems: A Systematic Literature ReviewPreprint
Mbaka, W., & van Gerwen, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4799844 · Cited by 1
Impacts of Photovoltaic Solar Energy on Soil Carbon: A Global Systematic Review and FrameworkPreprint
Krasner, N. Z., Fox, J., Armstrong, A., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4846467 · Cited by 2
Current Methodologies of Creating Material Passports: A Systematic Literature ReviewPreprint
Markou, I., Sinnott, D., & Thomas, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4893214 · Cited by 2
Sustainable Engineering of Machine Learning-Enabled Systems: A Systematic Mapping StudyPreprint
Chadli, K., Botterweck, G., & Saber, T. (2024) · Research Square · doi:10.21203/rs.3.rs-4694122/v1 · Cited by 8
Vergallo, R., Cruz, L., Errico, A., & Mainetti, L. (2024) · Research Square · doi:10.21203/rs.3.rs-4747830/v1 · Cited by 3
A Critical Analysis of Spoofing and Jamming Approaches in UAV NavigationPreprint
Meheretu, S. E., Nigussie, E., Gebremeskel, G. B., & Hailesilassie, S. Y. (2024) · SSRN Electronic Journal · doi:10.21203/rs.3.rs-5370811/v1 · Cited by 4
Architectural Degradation: Definition, Motivations, Measurement Andremediation ApproachesPreprint
Ahmed, N., Su, R., Esposito, M., Janes, A., Lenarduzzi, V., & Taibi, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5370705 · Cited by 3
Automatic unit test amplification for DevOps
Danglot, B. (2019) · theses.fr (ABES) · doi:10.70675/5747c493z5931z462bz8c5az0f897179fd20 · Cited by 2
“Specifically for the field of software engineering, well-known methodologies are systematic literature reviews (SLR) [Kitchenham 2004], systematic mapping studies (SMS) [Petersen 2008] and snowballing studies [Wohlin 2014].”
MethodMaxwell, G. S. (2021) · The enabling power of assessment · doi:10.1007/978-3-030-63539-8_1 · Cited by 1
Software Engineering for Internet of Things: The Practitioners' Perspective
Fahmideh, M., Ahmad, A., Behnaz, A., Grundy, J., & Susilo, W. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3070692 · Cited by 30
“Following with forward and backward search recursively for each identified paper’s references via snowballing [32], 53 new papers were added to our study list.”
MethodSaade, M. R. M., da Silva, M. G., & da Silva, V. G. (2017) · PARC Pesquisa em Arquitetura e Construção · doi:10.20396/parc.v8i4.8650295 · Cited by 2
“Systematic literature reviews (SLR), widely used in the medical sciences, stand out as a way of synthesizing evidence and allowing researchers to really grasp the status of a research area (WOHLIN, 2014).”
BackgroundHow to elicit and specify software requirements from BPMN diagrams?
Sorgatto, D. W., Paiva, D. M. B., & Cagnin, M. I. (2018) · doi:10.1145/3229345.3229403 · Cited by 1
“Deste último estudo percebeu-se a necessidade do uso da técnica snowballing backward, que visa pesquisar as referências relevantes que constam nas bibliografias dos trabalhos retornados pela string de busca utilizada na pesquisa que resultou no start set1 e foi realizado com o objetivo de enriquecer a busca por trabal…”
Team Topologies in Software Teams: A Multivocal Literature Review
Ahmed, W., & Colomo‐Palacios, R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87013-3_21 · Cited by 1
Akresh, M. E., Lamonde, S., Stokes, L., et al. (2022) · Journal of Caribbean Ornithology · doi:10.55431/jco.2022.35.29-39 · Cited by 1
Carvalho, L. C. D. S., Garcia, A., Assunção, W. K. G., et al. (2022) · Springer eBooks · doi:10.1007/978-3-031-11686-5_19 · Cited by 1
Shiraishi, T. (2023) · Journal of International Cooperation in Education · doi:10.1108/jice-07-2022-0016 · Cited by 1
Blockchain-Based Authentication in IoT Environments: A Survey
Alani, M. M. (2022) · Advances in information security · doi:10.1007/978-3-031-25506-9_5 · Cited by 1
Andrade, B. P. B., & Piran, F. S. (2023) · Anais ... Encontro Nacional de Engenharia de Produção/Anais do Encontro Nacional de Engenharia de Produção · doi:10.14488/enegep2023_tn_wpg_407_2003_46437 · Cited by 1
de Oliveira, K. P. R., da Silva, M. H. B., & Granja, A. D. (2023) · Simpósio Brasileiro de Gestão e Economia da Construção/Anais do ... Simpósio Brasileiro de Gestão e Economia da Construção · doi:10.46421/sibragec.v13i00.2619 · Cited by 1
Santo, L. A. A. D. E., de Castro, R. M. L., & Barros, R. D. S. (2023) · Revista Brasileira de Análise do Comportamento · doi:10.18542/rebac.v19i2.15667 · Cited by 1
A Review on Analytic Anomaly Detection of Fluid in Pipeline Using Machine Learning
Adamu, I. A., Zambuk, F. U., Ya’u, B. I., & Hussaini, H. (2024) · International Journal of Computer Science and Mobile Computing · doi:10.47760/ijcsmc.2024.v13i03.005 · Cited by 2
Lima, M. Y., SILVA, C. V. D., & Souza, R. (2022) · Workshop em Engenharia de Requisitos (WER) · doi:10.29327/1298262.25-23 · Cited by 1
“Realizamos um estudo bibliográfico sobre "qualidade da especificação de testes em projetos que usam BDD"e para isso, utilizamos a técnica de snowballing [27] [16] buscando identificar trabalhos sobre o assunto a partir de um conjunto inicial que possuía no seu título e abstract as palavras-chaves "acceptance testing",…”
BackgroundUnnecessary organizational burden: a conceptual framework
Ginzburg, O., Weil, S., & van Witteloostuijn, A. C. (2024) · Future Business Journal · doi:10.1186/s43093-024-00330-8 · Cited by 2
“Employing a meticulous snowball approach, we heed Wohlin’s [78] guidance, utilizing reference lists and citations to iteratively unearth additional relevant literature.”
MethodBidding for B2B or B2G tenders: toward the adoption of pricing models in practice
Herrmann, H., & Cobo, M. J. (2024) · Management Review Quarterly · doi:10.1007/s11301-024-00424-1 · Cited by 6
“Journal articles, conference papers, books, chapters, dissertations, reports, and trade journals were considered in a snowball procedure (Wohlin 2014).”
Towards Lenses for Reviewing Playfulness in HCI
França, N. C., Maurer, B., & Bertran, F. A. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650888 · Cited by 3
“The terms in the search query were snowballed from an unpub-lished scoping review on playfulness in HCI performed by the first author on the ACM Digital Library and Google Scholar [69].”
BackgroundOntoReD: requirements debt ontology
Bonfim, V. D., & Benitti, F. B. V. (2026) · Requirements Engineering · doi:10.1007/s00766-026-00459-8 · Cited by 1
“To address the state-of-the-art, the researchers conducted a systematic literature mapping (SLM), supported by the For - ward Snowballing Technique [ 28 ] to update the SLM, and a tertiary study.”
InfluentialFuture Directions in Defence NLP: Investigating Research Gaps for Low-Resource Languages
Teze, V., & Nazaruka, Ē. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63543-4_7 · Cited by 4
Characterization of ground-based telescope control systems: A systematic mapping study
Carrasco, S., Galeas, P., & Cravero, A. (2024) · Astronomy and Computing · doi:10.1016/j.ascom.2024.100854 · Cited by 3
An LGPD Compliance Inspection Checklist to Assess IoT Solutions
Neto, I. P. G., Mendes, J., Ferreira, W., et al. (2024) · doi:10.1145/3663529.3663853 · Cited by 1
Understanding tourism in social farming as a form of social innovation
Ferrara, A. (2024) · ECONOMIA AGRO-ALIMENTARE · doi:10.3280/ecag2024oa15793 · Cited by 1
Júnior, J. M. L. D. M., Viana, W., & Façanha, A. R. (2024) · Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia) · doi:10.5753/webmedia_estendido.2024.244084 · Cited by 2
Pricing Data Based on Value: A Systematic Literature Review
Radić, M., Herrmann, P., Stein, T., Heirich, N., & Radić, D. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-73125-9_20 · Cited by 6
Student teachers’ self-assessment during teaching practicum: A literature review
Pérez-Torregrosa, A.-B., & Arrufat, M. J. G. (2024) · AtoZ novas práticas em informação e conhecimento · doi:10.5380/atoz.v13i0.92347 · Cited by 2
“Snowballing incorporated an additional study (Wohlin, 2014) that met the IC.”
Smart cities policies for urban development: Systematic insight into public value creation
Barbieri, R., Natale, F., Cattivelli, V., & Coluccia, B. (2024) · Journal of Infrastructure Policy and Development · doi:10.24294/jipd9572 · Cited by 2
“Based on a brainstorming process and through a snowballing approach (Wohlin, 2014), two categories of keywords were selected.”
Review-Pulse: A Dashboard for Managing User Feedback for Android Applications
Abdelaziz, O., Codabux, Z., & Schneider, K. A. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00095 · Cited by 1
“These categories emerged after manually coding the app reviews in a snowball fashion [5], meaning we repeatedly labeled reviews until no new codes emerged, reaching saturation.”
Quality of Big Data Systems: a Systematic Review of Practices Methods and Tools
de Oliveira, I. S., Alves, J. M., Alcântara, S., Santos, I. D. S., & Andrade, R. M. C. (2024) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3701625.3701642 · Cited by 3
“To assess the quality of the included papers, we defined a quality assessment form [51] .”
Critical Factors for Software Product Management: A Systematic Literature Review
Paranhos, I., & Santos, G. (2024) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3701625.3701655 · Cited by 4
Analysis of Behavior-Driven Development: A Thematic Synthesis
Arredondo-Reyes, V. M., Domínguez-Isidro, S., Sánchez-García, Á. J., & Ocharán-Hernández, J. O. (2024) · Programming and Computer Software · doi:10.1134/s0361768824700713 · Cited by 4
Queering Software Engineering Education: Integrative Approaches and Student Experiences
Browning, J. W., Bustard, J., & Anderson, N. J. (2024) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet61869.2024.10837614 · Cited by 1
“A snowballing approach [15] was not utilized to add papers to our analysis, as this approach could potentially lead to papers published outside of IEEE Xplore and ACM.”
Posterior Urethral Valves and Fertility: Insight on Paternity Rates and Seminal Parameters
Corte, M. D., Nappo, S. G., Aversa, A., Vignera, S. L., Porpiglia, F., Fiori, C., & Mondaini, N. (2025) · Diseases · doi:10.3390/diseases13010021 · Cited by 4
“The bibliographic references of the selected works were then evaluated for potential snowball sampling or citation chaining [15].”
Belinski, V., Neto, A. G. S. S., & Emer, M. C. F. P. (2025) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2025.4103 · Cited by 1
de Oliveira, D. A. C., Ferreira, R. O. M., Picado-Loaiza, S., et al. (2025) · Trends in Anaesthesia and Critical Care · doi:10.1016/j.tacc.2025.101543 · Cited by 1
Tennant, R., & Randall, T. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3554969 · Cited by 4
“Relevant articles were also identified through hand-searching references using the snowballing technique, a popular method in systematic literature reviews for software engineering [31].”
Šklebar, J., Martinec, T., Škec, S., & Štorga, M. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.104805 · Cited by 5
Conflicting Perceptions of Project Success and Failure Relationships: A Systematic Literature Review
Eaton, C., Glandon, K. L., Teper, T., Mesmer, B. L., & Ferreira, S. (2025) · Engineering Management Journal · doi:10.1080/10429247.2025.2492921 · Cited by 1
“Sources referenced in included sources were searched in Google Scholar in a reverse (backward) snowballing method (Wohlin, 2014) and subjected to the same selection criteria in order to identify more sources than a traditional systematic review alone (Mourão et al., 2017; Wohlin, 2014; Wohlin et…”
AI-Powered Social Engineering and Impersonation Attacks
Arafah, M. E., Karadsheh, L., Aburub, F. A. F., & Alhariri, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-0832-6.ch006 · Cited by 3
Daemei, A. B., Lovreglio, R., Feng, Z., & Paes, D. (2025) · Lecture notes in civil engineering · doi:10.1007/978-981-96-1181-2_30 · Cited by 2
The Ephemeral Threat: Assessing the Security of Algorithmic Trading Systems powered by Deep Learning
Rizvani, A., Apruzzese, G., & Laskov, P. (2024) · ACM Conference on Data and Application Security and Privacy (CODASPY) · doi:10.1145/3714393.3726490 · Cited by 5
A Systematic Review of Common Beginner Programming Mistakes in Data Engineering
Neuwinger, M., & Riehle, D. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00024 · Cited by 3
“Throughout the process, we perform backward and forward snowballing to identify additional relevant articles [29].”
Food waste factors in university restaurants: evidence from Brazilian students
de Campos, E. A. R., Günther, W. M. R., Ribeiro, J. L. D., de Paula, I. C., & ten Caten, C. S. (2025) · International Journal of Sustainability in Higher Education · doi:10.1108/ijshe-08-2024-0581 · Cited by 3
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.