Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 1,801–1,900 of 3,934
Leveraging encoder-only large language models for mobile app review feature extraction
Motger, Q., Miaschi, A., Dell’Orletta⋄, F., Franch, X., & Marco, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10660-y · Cited by 18
“For the elicitation of threats to validity, we rely on the taxonomy proposed by Wohlin (2014).”
Choe, K., Bakker, L., van den Hove, D., Eussen, S. J. P. M., Kenis, G., Ramakers, I. H. G. B., Verhey, F. R. J., Rutten, B. P. F., & Köhler, S. (2025) · GeroScience · doi:10.1007/s11357-025-01636-3 · Cited by 13
“Additionally, as part of a snowballing approach, reference lists of selected studies were screened, and some authors were contacted to provided extra references [32].”
Best Practices Evidenced for Software Development Based on DevOps and Scrum: A Literature Review
Pardo, M. A. P., Ordóñez, H., Cobos, C., & Muñoz, M. (2025) · Applied Sciences · doi:10.3390/app15105421 · Cited by 17
Towards the Deployment of Realistic Autonomous Cyber Network Defence: A Systematic Review
Vyas, S., Mavroudis, V., & Burnap, P. (2025) · ACM Computing Surveys · doi:10.1145/3729213 · Cited by 28
“Due to the area of ACND gaining popularity only recently, backward snowballing [68] and forward snowballing [134] were conducted for several searches in order to find publications and code repositories relevant to ACND that were not listed in the search strategy.”
A Systematic Review of Cyber Range Taxonomies: Trends, Gaps, and a Proposed Taxonomy
Lillemets, P. S., Jawad, N. B., Kashi, J., Sabah, A., & Dragoni, N. (2025) · Future Internet · doi:10.3390/fi17060259 · Cited by 22
A Survey on Immersive Cyber Situational Awareness Systems
Ahmad, H., Ullah, F., & Jafri, R. (2025) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp5020033 · Cited by 26
A review of lead exposure source attributional studies
Kinally, C., Fuller, R., Larsen, B., Hu, H., & Lanphear, B. P. (2025) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2025.179838 · Cited by 16
External Validation of AI Models for Skin Diseases: A Systematic Review
Fernandes, T. R. S., Teles, A. S., Fernandes, J. R. N., Lima, L. D. B., Sousa, D. L., Soares, R. D. C., & Teixeira, S. S. (2025) · IEEE Access · doi:10.1109/access.2025.3584904 · Cited by 13
“Finally, we applied the backward and forward snowballing techniques [51], [96] to the initially selected studies to ensure comprehensive coverage of relevant literature.”
Psychological safety in software workplaces: A systematic literature review
de Santana, B. S., Monte, L., Silva, B. S. D. A., Carneiro, G. D. F., Freire, S., Santos, J. A. M., & Mendonça, M. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107838 · Cited by 27
“Snowballing is a method for identifying relevant sources by examining the references of discovered articles (Wohlin, 2014).”
Bucaioni, A., Salle, A. D., Iovino, L., Pelliccione, P., & Raimondi, F. (2025) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa65012.2025.00027 · Cited by 15
“We then conducted closed recursive backward and forward snowballing [48] to mitigate potential construct validity biases [25], but no additional sources were identified.”
Vision-Based Mobile App GUI Testing: A Survey
Yu, S., Fang, C., Tuo, Z., Zhang, Q., Chen, C., Chen, Z., & Su, Z. (2025) · ACM Computing Surveys · doi:10.1145/3773027 · Cited by 37
“To further expand our corpus and mitigate the limitations of keyword search, we apply snow-balling [236] to each paper included in the initial result set.”
Cognitive Hexagon-Controlled Intelligent Speech Interaction System
Chaurasiya, H. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3168807 · Cited by 12
“The most important in this method was the snowball technique [18] used with the following steps.”
MethodHow Can No/Low Code Platforms Help End-Users Develop ML Applications? - A Systematic Review
Li, L., & Wu, Z. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21707-4_25 · Cited by 23
A Microservice Deployment Guide
Nino-Martinez, V. M., Ocharán-Hernández, J. O., Limón, X., & Pérez-Arriaga, J. C. (2022) · Programming and Computer Software · doi:10.1134/s0361768822080151 · Cited by 13
Context, Content, Consent - How to Design User-Centered Privacy Explanations (S)
Brunotte, W., Droste, J., & Schneider, K. A. (2023) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2023-032 · Cited by 13
Do Current Online Coding Tutorial Systems Address Novice Programmer Difficulties?
Alasmari, O. A., Singer, J., & Ada, M. B. (2023) · International Conference on Education Technology and Computer · doi:10.1145/3629296.3629333 · Cited by 12
“, a snowballing technique was employed to uncover features of supportive systems that could potentially serve as solutions for the common programming learning difficulties identified in Section 4.1 [40].”
InfluentialVirtual burn care – Friend or foe? A systematic review
Mondor, E., Barnabe, J., Laguan, E. M. R., & Malic, C. (2024) · Burns · doi:10.1016/j.burns.2024.02.014 · Cited by 9
Identifying Authorship in Malicious Binaries: Features, Challenges & Datasets
Gray, J., Sgandurra, D., Cavallaro, L., & Blasco, J. (2024) · ACM Computing Surveys · doi:10.1145/3653973 · Cited by 19
“It also allowed us to use the backward and forward snowball technique [116] to systematically discover other relevant academic works focusing on papers published in top conferences.”
Method6G secure quantum communication: a success probability prediction model
Akbar, M. A., Khan, A. A., Hyrynsalmi, S., & Khan, J. A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00427-y · Cited by 14
“This strategy resulted in a continuously expanding sample size as we delved deeper into references and citations (Wohlin 2014).”
Nguyen, T. M. N., Brion, S., & Chauvet, V. (2024) · European Management Review · doi:10.1111/emre.12649 · Cited by 10
van Roosmale, S., Audenaert, A., & Meysman, J. (2024) · Facilities · doi:10.1108/f-05-2023-0042 · Cited by 12
Unmasking Data Secrets: An Empirical Investigation into Data Smells and Their Impact on Data Quality
Recupito, G., Rapacciuolo, R., Nucci, D. D., & Palomba, F. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644960 · Cited by 12
Jacquet, L., Duigou, A. L., & Kerbrat, O. (2024) · Journal of Composites Science · doi:10.3390/jcs8070276 · Cited by 9
“This literature has been retrieved by a snowballing method [17] described in a conference paper from V é ronique Michaud [18].”
MethodA Survey on Penetration Path Planning in Automated Penetration Testing
Chen, Z., Kang, F., Xiong, X., & Shu, H. (2024) · Applied Sciences · doi:10.3390/app14188355 · Cited by 26
Modelling language quality evaluation in model-driven information systems engineering: A roadmap
Giraldo, F. D., España, S., Giraldo, W. J., & Pastor, Ó. (2015) · International Conference on Research Challenges in Information Science (RCIS) · doi:10.1109/rcis.2015.7128864 · Cited by 9
“the snowballing procedure reported in [4].”
Features of IT Service Markets: A Systematic Literature Review
Jazayeri, B., Platenius, M. C., Engels, G., & Kundisch, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46295-0_19 · Cited by 10
Security and Privacy Concerns in Connected Cars: A Systematic Mapping Study
Ram, P., Markkula, J., Friman, V., & Raz, A. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00029 · Cited by 12
“A total of 93 primary studies were selected, which was supplemented by adopting backward snowballing [18].”
BackgroundDesigning Model of Serious Game for Flood Safety Training
Mokhtar, N., Ismail, A., & Muda, Z. (2019) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2019.0100541 · Cited by 14
“Meanwhile, another approach used is called snowballing [16] (i.e. retrieving relevant papers based on target papers‟ references list or paper citing).”
MethodA Research Landscape on Formal Verification of Software Architecture Descriptions
Araújo, C., Cavalcante, E., Batista, T., et al. (2019) · IEEE Access · doi:10.1109/access.2019.2953858 · Cited by 9
“Third, we have not performed snow-balling [26], a technique that allows checking the reference lists of the read studies aiming to find additional studies not retrieved in the automated search procedure.”
Salgado, A. D. L., Fortes, R. P. D. M., de Oliveira, R. R., & Freire, A. P. (2020) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2020.100984 · Cited by 10
Website disclosure of environmental compliance management—the case of European production companies
Thimm, H. H., & Rasmussen, K. B. (2020) · Journal of Environmental Studies and Sciences · doi:10.1007/s13412-020-00643-4 · Cited by 10
“A snowballing approach was performed based on the guidelines of Wohlin (2014).”
MethodJogos Sérios no Processo de Ensino e Aprendizagem de Interação Humano-Computador
de Sales, A. B., & Silva, M. A. S. E. (2020) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/cbie.sbie.2020.552 · Cited by 9
“O Snowballing consiste em avaliar a lista de referências e citações de um conjunto de estudos primários relevantes para identificar outros estudos relevantes [Wohlin 2014].”
BackgroundSidani, A., Duarte, J., Baptista, J. S., et al. (2021) · International Journal of Occupational and Environmental Safety · doi:10.24840/2184-0954_005.001_0003 · Cited by 9
Identity in K-12 Computer Education Research: A Systematic Literature Review
Große-Bölting, G., Gerstenberger, D., Gildehaus, L., Mühling, A., & Schulte, C. (2021) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3446871.3469757 · Cited by 14
“Therefore, even before the search was started, a process for forward and backward snowballing [117] was established – as a first adjustment to the classic [65] process.”
MethodAssessing the outcomes of implementing natural open space plans in a Global South city
Wessels, N., Sitas, N., O’Farrell, P. J., & Esler, K. J. (2021) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2021.104237 · Cited by 11
VALUE APPROPRIATION AND INNOVATION COLLABORATION DYNAMICS: A REVIEW AND RESEARCH AGENDA
Yang, J., Hurmelinna‐Laukkanen, P., Sharma, A., & Westerlund, M. (2021) · International Journal of Innovation Management · doi:10.1142/s1363919621400077 · Cited by 12
Russo, M., Angelosanti, M., Bernardini, G., et al. (2021) · Sustainability · doi:10.3390/su14010042 · Cited by 9
“To include relevant documents not identified with the described procedure, a final snowball search [37] was conducted among the references of the final studies found through the systematic process, applying the same selection criteria used for the systematic screening (Figure 1).”
MethodSensitivity of Physiological Measures of Acute Driver Stress: A Meta-Analytic Review
Kerautret, L., Dabic, S., & Navarro, J. (2021) · Frontiers in Neuroergonomics · doi:10.3389/fnrgo.2021.756473 · Cited by 15
“In addition, a snowballing approach (Wohlin, 2014) was used to retrieve additional references.”
MethodDigital Contact Tracing Apps for COVID-19: Development of a Citizen-Centered Evaluation Framework
Tsvyatkova, D., Buckley, J., Beecham, S., Chochlov, M., O’Keeffe, I. R., Razzaq, A., Rekanar, K., Richardson, I., Welsh, T., Storni, C., et al. (2022) · JMIR mhealth and uhealth · doi:10.2196/30691 · Cited by 12
“Consulting the reference lists of the selected articles to identify further relevant studies, following the systematic “backward snowballing” protocol proposed by Wohlin et al [23].”
Enhancing the Replication Potential of Smart Lighting Projects
Talmar, M., Romme, A. G. L., & Valkenburg, R. (2022) · Smart Cities · doi:10.3390/smartcities5020032 · Cited by 11
Familial Male-limited Precocious Puberty (FMPP) and Testicular Germ Cell Tumors
Kooij, C. D., Mavinkurve‐Groothuis, A. M. C., Hovinga, I. C. L. K., Looijenga, L. H. J., Rinne, T., Giltay, J. C., de Kort, L. M. O., Klijn, A. J., de Krijger, R. R., & Stuart, A. A. V. (2022) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/clinem/dgac516 · Cited by 8
Sigahi, T. F. A. C., & Sznelwar, L. I. (2023) · Kybernetes · doi:10.1108/k-11-2022-1507 · Cited by 13
Treatment with anticancer drugs for advanced pancreatic cancer: a systematic review
Salazar, J., Bracchiglione, J., Savall-Esteve, O., Antequera, A., Bottaro, D., Gutiérrez‐Valencia, M., Martínez-Peralta, S., Pericay, C., Tibau, A., Bonfill, X., et al. (2023) · BMC Cancer · doi:10.1186/s12885-023-11207-4 · Cited by 10
Security Challenges and Solutions in theInternet of Things
Porras, J., Khakurel, J., Knutas, A., & Pänkäläinen, J. (2018) · Nordic and Baltic Journal of Information and Communications Technologies · doi:10.13052/nbjict1902-097x.2018.010 · Cited by 9
A Multi-concern Method for Identifying Business Services: A Situational Method Engineering Study
Adalı, O. E., Türetken, O., Özkan, B., Gilsing, R., & Grefen, P. (2020) · Lecture notes in business information processing · doi:10.1007/978-3-030-49418-6_15 · Cited by 12
“Next, we applied backward snowballing on these studies [17].”
MethodSafety Case Maintenance: A Systematic Literature Review
Cârlan, C., Gallina, B., & Soima, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-83903-1_8 · Cited by 13
Software Testing Process Models Benefits & Drawbacks: a Systematic Literature Review.Preprint
Hrabovská, K., Rossi, B., & Pitner, T. (2019) · arXiv (Cornell University) · Cited by 9
Zhu, M., & Wang, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4684304 · Cited by 8
Software Module Clustering: An In-Depth Literature Analysis
Sarhan, Q. I., Ahmed, B. S., Bureš, M., & Zamli, K. Z. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3042553 · Cited by 52
“The snowballing [23] search method was applied to the remaining papers to reduce the possibility of missing critical related papers.”
MethodHolly, F., Schild, C., & Schlund, S. (2024) · Circular Economy · doi:10.55845/xupf2540 · Cited by 14
“To guarantee the most comprehensive and complete search possible at the time of the search, a final snowballing was also added, according to Wohlin (2014).”
Towards Quality Assurance in MOOCs: A Comprehensive Review and Micro-Level Framework
Sebbeq, H., & Faddouli, N.-E. E. (2024) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v25i1.7544 · Cited by 21
“In this section, we detail the research methodology used for this literature review, following Kitchenham’s guidelines (Kitchenham et al., 2010) and incorporating the so-called snowballing procedure proposed by Wohlin (2014).”
MethodKurtzhals, M., Bjerregaard, A., Hybschmann, J., Müllertz, A. L. O., DeSilva, B., Elsborg, P., Timm, A., Petersen, T. L., Thygesen, L. C., Kurtzhals, P., et al. (2024) · Obesity Reviews · doi:10.1111/obr.13742 · Cited by 15
Robustness of models addressing Information Disorder: A comprehensive review and benchmarking study
Fenza, G., Loia, V., Stanzione, C., & Gisi, M. D. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127951 · Cited by 12
Bilius, L.-B., Schipor, O.-A., & Vatavu, R.-D. (2024) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3639701.3656327 · Cited by 11
Hassan, S., Li, Q., Aurangzeb, K., Yasin, A., Khan, J. A., & Anwar, M. S. (2024) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12348 · Cited by 10
“The snowball [61] strategy was also applied to all studies to detect research articles that are associated with our research topic.”
Systematic Literature Review of Commercial Participation in Open Source Software
Li, X., Zhang, Y., Osborne, C., Zhou, M., Jin, Z. G., & Liu, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3690632 · Cited by 41
Hosseini, M. M., Koohpaei, A., Ebrahimipour, H., & Hosseini, S. T. M. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2024.102875 · Cited by 21
“Following Wohlin ’ s guidelines, 43 a snowball method was conducted for further relevant studies.”
MethodDecentralized autonomous organization in built environments: applications, potential and limitations
Ly, R., & Shojaei, A. (2025) · Information Systems and e-Business Management · doi:10.1007/s10257-025-00699-1 · Cited by 24
“To ensure a comprehensive review, the snowballing techniques (Wohlin 2014) were also used to identify relevant papers that hadn’t been found through the traditional database search.”
Transfer line balancing problem: A comprehensive review, classification, and research avenues
Beldar, P., Fathi, M., Nourmohammadi, A., et al. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.110913 · Cited by 8
“To ensure comprehensive coverage of the TLBP literature, a snow-ball search approach was employed to supplement the initial database search (Wohlin, 2014).”
Martín-Olmedo, J. J., & Jurado‐Fasoli, L. (2025) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2025.2472858 · Cited by 9
“In total, 40 studies were included in this work, and additionally, 13 studies citing the previously included ones were incorporated ( forward snowballing search ) (Wohlin 2014).”
Madhuhansi, W. H. T., Ozanne, L. K., Kennedy, A., & Kemper, J. A. (2025) · International Journal of Consumer Studies · doi:10.1111/ijcs.70032 · Cited by 17
Wysocki, W., Miciuła, I., & Plecka, P. (2025) · Electronics · doi:10.3390/electronics14071331 · Cited by 13
Cognitive Factors in Process Model Comprehension—A Systematic Literature Review
Möller, M., Winter, M. R., & Reichert, M. (2025) · Brain Sciences · doi:10.3390/brainsci15050505 · Cited by 26
“To expand the dataset, backward and forward snowballing techniques are employed [40]—backward snowballing reviews the references of selected studies, whereas forward snowballing identifies studies that cite them.”
SME Business Agility Framework: A Comprehensive Review of Capabilities, Dimensions, and Enablers
Arno, P. (2025) · IEEE Access · doi:10.1109/access.2025.3572526 · Cited by 20
“As suggested by [71], we repeated this process iteratively until no new relevant studies were identified.”
Mantalovas, S., Karakousis, V. A., Sevva, C. D., Roulia, P., Savvakis, S., Anthimidis, G., Papadopoulos, K., Magra, V., Varsamis, N., Kosmidis, C. S., et al. (2025) · Cancers · doi:10.3390/cancers17121934 · Cited by 12
“Additionally, a thorough examination of the reference lists of the selected studies was performed to identify potentially relevant studies using the snowball effect methodology [49].”
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107802 · Cited by 35
Ameer, S., Muhammad, N., Jadoon, K. Z., Sheikh, N., & Rahman, K. U. (2025) · Applied Water Science · doi:10.1007/s13201-025-02488-0 · Cited by 12
“In snowballing the literature is collected using either citations or the references of the research papers or both (Wohlin et al. 2014).”
Dost, B., CASSAI, A. D., Bugada, D., Balzani, E., Karapınar, Y. E., Beldağlı, M., Yalin, M. S. O., Tulgar, S., & Ahişkalıoğlu, A. (2025) · Minerva Anestesiologica · doi:10.23736/s0375-9393.25.18771-3 · Cited by 9
On‐Table Extubation After Pediatric Cardiac Surgery: A Systematic Review
Tapioca, V. D. O., Caetano, L., Gibicoski, T., et al. (2025) · Pediatric Anesthesia · doi:10.1111/pan.70026 · Cited by 8
Papile, F., & Curto, B. D. (2025) · Sustainability · doi:10.3390/su17198804 · Cited by 8
Kosenkov, O., Zabardast, E., Fucci, D., Méndez, D., & Unterkalmsteiner, M. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107946 · Cited by 16
“Based on the initial seed of the relevant primary studies, we conducted forward and backward snowballing [53] using the same selection criteria.”
Müller, J., & Wagner, I. (2025) · European Physical Education Review · doi:10.1177/1356336x251357207 · Cited by 43
“An additional three studies that were not extracted after using the search string were identi fi ed using the snowballing method (Wohlin, 2014); this method involved identifying relevant studies by examining the reference lists of key articles to expand the pool of relevant literature systematically…”
Wubineh, B. Z., Deriba, F. G., & Gemeda, F. W. (2026) · BMC Medical Ethics · doi:10.1186/s12910-026-01404-8 · Cited by 11
“To make our study robust, we included additional studies ( N = 7) through backward and forward snowballing [36].”
Spennemann, D. H. R. (2020) · Plant Ecology & Diversity · doi:10.1080/17550874.2020.1819465 · Cited by 11
“In consequence, the methods espoused a combination of systematic database searches with snowballing (Atkinson and Flint 2001; Greenhalgh and Peacock 2005; Wohlin 2014).”
Physiological Monitoring Systems for Firefighters (A Short Review)
Bustos, D., Guedes, J. C., Baptista, J. S., Vaz, M. A. P., Costa, J. T., & Fernandes, R. J. (2021) · Studies in systems, decision and control · doi:10.1007/978-3-030-89617-1_27 · Cited by 9
Mobile Gamification Applied to Employee Productivity in Companies: A Systematic Review
Miranda, M. A. C., & Vergaray, A. D. (2021) · TEM Journal · doi:10.18421/tem104-50 · Cited by 14
“Organizations take into account that employees are currently linked to a mobile device and make constant use of applications that help them relax and be entertained [50].”
BackgroundAccessibility in the Software Development Life Cycle: A Systematic Literature Review
Cruz-Portilla, M., Pérez-Arriaga, J. C., Ocharán-Hernández, J. O., & Sánchez-García, Á. J. (2021) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft52520.2021.00024 · Cited by 11
Physiology-based personalization of persuasive technology: a user modeling perspective
Spelt, H., Westerink, J. H. D. M., Frank, L. E., Ham, J. R. C., & IJsselsteijn, W. A. (2022) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-021-09313-8 · Cited by 11
“For finding relevant research, this paper used a backward and forward snowballing procedure (Wohlin 2014).”
MethodUniversiti Teknologi, M. A. R. A., Samsudin, N. S., Khalil, N., et al. (2022) · International Journal of Sustainable Construction Engineering Technology · doi:10.30880/ijscet.2022.13.02.004 · Cited by 8
API Deprecation: A Systematic Mapping Study
Bonorden, L., & Riebisch, M. (2022) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa56994.2022.00076 · Cited by 9
“Furthermore, we applied forward and backward snowballing to the identified primary studies [23]: First, we collected the references and citations using the Semantic Scholar Academic Graph API [24].”
MethodDeep Learning-Based Code Refactoring: A Review of Current Knowledge
Naik, P., Nelaballi, S., Pusuluri, V. S., & Kim, D. (2023) · Journal of Computer Information Systems · doi:10.1080/08874417.2023.2203088 · Cited by 26
“…we present a systematic literature review on the application of DL to code refactoring, following the guidelines by Kitchenham et al. 25 We also used the “snow-balling” technique 26 to identify additional sources in the field, by including the references used in other relevant survey papers.”
MethodMantau, M. J., & Benitti, F. B. V. (2023) · doi:10.5753/sbsc.2023.229064 · Cited by 9
“The assessment instruments were developed us-ing the GQM approach [Basili 1992], and guidelines presented by [Wohlin et al. 2012], [Wohlin 2014], and [DeVellis 2016].”
MethodLCA in Saudi Arabia: a critical review
Shaukat, M. M. (2023) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-023-02197-3 · Cited by 8
“In addition, using snowballing by checking the citations in selected papers can help to find more papers pertinent to the research area (Wohlin 2014).”
MethodTerzimehić, N., Draxler, F., Ahsanpour, M., & Schmidt, A. (2023) · Mensch und Computer (MuC) · doi:10.1145/3603555.3603578 · Cited by 10
“To this end, we followed a snowballing approach [84].”
MethodCognitive personalization for online microtask labor platforms: A systematic literature review
Paulino, D., Correia, A., Barroso, J., & Paredes, H. (2023) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09383-w · Cited by 14
“After collecting the primary studies eligible for this SLR, we also used snowballing techniques to obtain more potentially relevant studies (Wohlin 2014).”
Obarafor, V., Qi, M., & Zhang, L. (2023) · doi:10.1109/icsip57908.2023.10270935 · Cited by 14
“The advantage of utilizing the snowball technique is that it helps researchers in obtaining significant research papers that are in-line with the research issues, solutions as well as the cited papers in a research work [9].”
BackgroundTactics for Software Energy Efficiency: A Review
Balanza-Martinez, J., Lago, P., & Verdecchia, R. (2024) · Progress in IS · doi:10.1007/978-3-031-46902-2_7 · Cited by 10
Thite, M., & Iyer, R. (2024) · Personnel Review · doi:10.1108/pr-04-2023-0358 · Cited by 15
Rüdele, K., Wolf, M., & Ramsauer, C. (2024) · Management of Environmental Quality An International Journal · doi:10.1108/meq-07-2023-0195 · Cited by 14
The Utilization of 3D Printers by Elementary-Aged Learners: A Scoping Review
Fokides, E., & Lagopati, G. (2024) · Journal of Information Technology Education Innovations in Practice · doi:10.28945/5288 · Cited by 21
“This method involves a thorough analysis of the reference lists from the collected papers, thereby uncovering additional studies that may have eluded initial search efforts (Wohlin, 2014).”
MethodLarcipretti, A. L. L., Gomes, F. C., Udoma‐Udofa, O. C., Dagostin, C. S., de Oliveira, J. S., Nager, G. B., Pontes, J. P. M., & Bannach, M. D. A. (2024) · Seizure · doi:10.1016/j.seizure.2024.07.017 · Cited by 8
Securing Electric Vehicle Charging Stations: A Critical Analysis of Authentication Vulnerabilities
Stichow, A., & Rempel, P. (2024) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew61692.2024.00037 · Cited by 8
“To systematically search for literature, we applied the snowball sampling method [35] and used the works of Alcaraz et al. [4] and Garofalaki et al. [17] as our starting point, along with the requirement for authentication mechanisms to be supported for OCPP 2.0.1 as inclusion criteria.”
MethodInfluentialGamification solutions for persons with disabilities: a systematic literature review
Boubakri, M., & Nafil, K. (2024) · Universal Access in the Information Society · doi:10.1007/s10209-024-01170-7 · Cited by 14
“Alongside the steps outlined above, we implemented a snowballing technique, a method employed to uncover additional pertinent articles by examining the reference lists of the identified articles [39].”
Central Bank Digital Currency: A Multivocal Literature Review
Silva, E. C., & da Silva, M. M. (2025) · Journal of Internet and Digital Economics · doi:10.1108/jide-03-2024-0013 · Cited by 13
Understanding the Board’s Involvement in Information Technology Governance
Matta, M., Cavusoglu, H., & Benbasat, I. (2022) · Information Systems Management · doi:10.1080/10580530.2022.2074580 · Cited by 8
“A key benefit of this approach is that because its focus is on papers referenced or on papers citing the papers included in the sample, the possibility of noise in the data sample is less compared to a database approach (Wohlin, 2014).”
BackgroundTurki, H. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.002 · Cited by 8
A Review of Design Interventions for Promoting Adolescents' Physical Activity
Ma, Y., Veldhuis, A., Bekker, T. M., Hu, J., & Vos, S. B. (2019) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3311927.3323130 · Cited by 9
“4) A snowballing procedure with a forward search [53] in the reference list of 22 selected papers was performed, resulting in three extra papers [6,31,34].”
MethodA Set of Usability Heuristics for Mobile Applications
da Costa, R. P., & Canedo, E. D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22646-6_13 · Cited by 17
Pragmatic interoperability in IoT: a systematic mapping study
Muniz, M. H. D. S., David, J., Braga, R. M. M., Campos, F., & Ströele, V. (2019) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3323503.3349561 · Cited by 9
“Tais métodos servem, respectivamente para pesquisar artigos que foram citados ( Backward ) e que citam ( Forward ) os estudos primários [28].”
Background
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.