Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 2,601–2,700 of 3,934
Reshaping Business Models with an Environmental Perspective
Sánchez-Planelles, J., & Segarra‐Oña, M. (2018) · Ecoproduction · doi:10.1007/978-3-642-33851-9_8 · Cited by 3
Face Recognition in Unconstrained Conditions: A Systematic Review
Shepley, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04404 · Cited by 10
“trategy referred to as snowballing was also employed. Snowballing involves collecting a set of primary papers, then referring to the reference lists in these papers to identify further primary papers [60]. This process is repeated until all primary papers are found. It is an effective means of collecting all primary s…”
MethodMaker in Electrical Engineering Education Based on Emergent Technology: Mapping Study
Martínez, R. (2019) · IEEE Revista Iberoamericana de Tecnologias del Aprendizaje · doi:10.1109/rita.2019.2950137 · Cited by 4
““Snowballing” technique [37] was applied in this last phase: the reference lists of all fulltext papers were scanned to decide whether to include it or not.”
MethodA Review of Ethical Discussions on Platforms and Ecosystems.
Hyrynsalmi, S., Koskinen, J. S. S., & Hyrynsalmi, S. M. (2019) · Tampere University Institutional Repository (Tampere University) · Cited by 3
Incremental Development of Safety Cases: a Mapping Study
Almendra, C., Silva, C., & Vilela, J. (2020) · Brazilian Symposium on Software Engineering · doi:10.1145/3422392.3422398 · Cited by 5
Feature Toggle Driven Development: Practices usedby Practitioners.Preprint
Mahdavi-Hezaveh, R., Dremann, J., & Williams, L. A. (2019) · arXiv (Cornell University) · Cited by 3
Owusu, M. F. (2019) · UC Research Repository (University of Canterbury) · doi:10.26021/9439 · Cited by 8
Application Security Review Criteria for DevSecOps Processes
Heilmann, J. (2020) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 3
Teaching Machine Learning in K-12 Computing Education: Potential and PitfallsPreprint
Tedre, M., Toivonen, T., Kahila, J., Vartiainen, H., Valtonen, T., Jormanainen, I., & Pears, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11034 · Cited by 11
“Second, in order to extend the overview of the field, snowball sampling was applied to the bibliography sections of the articles already identified [111].”
MethodPyNose: A Test Smell Detector For PythonPreprint
Wang, T., Golubev, Y., Smirnov, O., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04639 · Cited by 3
“Thus, we conducted backward snowballing [34] (i.”
Gholamian, S., & Ward, P. A. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12489 · Cited by 43
“, a single round of backward and forward snowballing search [67] with Google”
BackgroundMethodHautopp, H. (2022) · VBN Forskningsportal (Aalborg Universitet) · doi:10.54337/aau469549955 · Cited by 9
Shojaifar, A. (2023) · Utrecht University Repository (Utrecht University) · doi:10.33540/1953 · Cited by 5
Data Preparation for Deep Learning Based Code Smell Detection: A Systematic Literature ReviewPreprint
Zhang, F., Zhang, Z., Keung, J., Tang, X., Yang, Z., Yu, X., & Hu, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4693778 · Cited by 3
IDENTIFYING CONCERNS WHEN SPECIFYING MACHINE LEARNING-ENABLED SYSTEMS: A PERSPECTIVE-BASED APPROACH
Villamizar, H. (2023) · doi:10.17771/pucrio.acad.65972 · Cited by 3
Chan, A., Jamieson, C., Draper, H., et al. (2024) · medRxiv · doi:10.1101/2024.04.17.24305969 · Cited by 3
Waseem, M., Ahmad, A., Liang, P., Akbar, M. A., Khan, A. A., Ahmad, I., Setälä, M., & Mikkonen, T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5151248 · Cited by 5
Friemert, D., Schnur, D., Runkel, S., Borsch, J., Karamanidis, K., Dellen, B. K., Thieme, L., Fiedler, A., Jaekel, U., & Hartmann, U. (2025) · medRxiv · doi:10.1101/2025.05.29.25328474 · Cited by 5
Requirements Engineering for Collaborative Artificial Intelligence Systems: A Literature Survey
Odong, L. A., Perini, A., & Susi, A. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-05760-1_24 · Cited by 7
Gutiérrez, L. F. R., Moreno, P. B., Suescún, E., Castro, J. L. A., & Pardo, C. (2023) · Heritage and Sustainable Development · doi:10.37868/hsd.v5i2.226 · Cited by 3
“In addition, to limit the potential of missing significant articles, we applied the snowballing technique [43] as a supplemental search strategy.”
MethodWhat Do We Know About Informal Caregiving in the Field of Addiction?: A Scoping Review
Côté, M., Morency, E.-E., Roussel, N., Ferland, F., Tremblay, J., & Blanchette-Martin, N. (2024) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-024-01301-4 · Cited by 2
“To identify as many relevant articles as possible, the final step was to apply the snowball method to the articles selected among both the database articles and the grey literature (Wohlin, 2014).”
From Creation to Exploitation: The Oracle Lifecycle
Eshghie, M., Jafari, M., & Artho, C. V. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c62648.2024.00009 · Cited by 2
“This study commenced with a thorough literature review according to snowballing guidelines by Wohlin [16].”
MethodCinquini, L., Pigatto, G., Tenucci, A., et al. (2024) · MANAGEMENT CONTROL · doi:10.3280/maco2024-002008 · Cited by 2
User Experience Techniques Adopted by Brazilian Game Development Studios: An Exploratory Study
da Silva, F. F., Leme, M. H. C., Zanella, A. F., et al. (2024) · International Journal of Computer Games Technology · doi:10.1155/2024/1857881 · Cited by 2
Potential Adverse Effects Caused by Gamification Elements in mHealth Apps
Rocha, I., & Silveira, M. S. (2024) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2024.4310 · Cited by 2
“To investigate the potential adverse effects of gamification in mHealth applications, a literature review was carried out using the snowballing procedure [Wohlin, 2014].”
InfluentialAgile User-Centered Development of Mobile Applications: A Systematic Literature Review
Fernández, A. C., Flores, I. A. R., & Estrella, A. L. P. (2024) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft63288.2024.00012 · Cited by 2
Analysing Coherence in Policy for Multi‐Functional Landscapes: An Exploratory Framework
Öhman, F., & Karlsson, M. (2025) · Environmental Policy and Governance · doi:10.1002/eet.2149 · Cited by 3
“This can be counteracted by selecting a strong initial set of studies that decreases bias (Wohlin 2014).”
Ahmadjee, S., Mera‐Gómez, C., Bahsoon, R., et al. (2025) · Distributed Ledger Technologies Research and Practice · doi:10.1145/3721140 · Cited by 2
“…our research focuses on the security risk assessment methodologies for blockchain adaption rather than on how blockchain technology is employed as a solution for use in risk management or as a method to provide security to applications such as blockchain-based Internet of Things (IoT) applications.”
Grannum, J., Siiman, L. A., & Tamm, A. (2024) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca62237.2024.10928113 · Cited by 3
“With a limited number of articles identified after reading the abstracts from the first search, snowballing as guided by Wohlin C. (2014) was used in an iterative cycle for the backward search of references and a forward citation.”
A Systematic Literature Review on Graphical User Interface Testing Through Software Patterns
Kousar, A., Khan, S. U. R., Mashkoor, A., & Iqbal, J. (2025) · IET Software · doi:10.1049/sfw2/9140693 · Cited by 4
“…we utilized two primary snowballing meth-ods: (i) backward snowballing, involving the exploration of reference lists from selected studies to discover new papers, and (ii) forward snowballing, which entailed analyzing paper citations via Google Scholar to identify the most relevant studies [39].”
Recruitment Strategies in App-Based Travel Surveys: Methodological Explorations
Svaboe, G. B. A., Tørset, T., & Lohne, J. (2021) · Transportation Research Part A Policy and Practice · doi:10.1016/j.tra.2025.104499 · Cited by 9
Viglio, S. S., Rodrigues, P. A., Jawabri, O. A., Espíndola, V. P., Figueredo, L. S., Bulcão-Neto, R. F., Ribeiro‐Rotta, R. F., & Neto, V. V. G. (2025) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi.2025.245959 · Cited by 2
Bokrantz, J., Subramaniyan, M., & Skoogh, A. (2025) · International Journal of Production Economics · doi:10.1016/j.ijpe.2025.109704 · Cited by 5
Context Conquers Parameters: Outperforming Proprietary Llm in Commit Message Generation
Imani, A., Ahmed, I., & Moshirpour, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00048 · Cited by 21
“Participant Recruitment : We adopted the snowball sampling approach to recruit the participants for our surveys [38].”
Gramaglia, C. M., Gambaro, E., Bestagini, L., et al. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327486 · Cited by 2
Degol, L., Olaerts, I., Jacobs, S., et al. (2025) · Journal of Wound Ostomy and Continence Nursing · doi:10.1097/won.0000000000001197 · Cited by 2
Tong, M., Paine, A. L., Evans, J., Weaterton, S., & Waters, C. S. (2025) · European Journal of Developmental Psychology · doi:10.1080/17405629.2025.2533160 · Cited by 4
“These must be assessed using a validated measure(s) (e.g., HADS, PHQ-9, GAD-7, PSI) Parental measures: Parental measures not specifically assessing parental mental health, psychological distress or parenting stress (Wohlin, 2014).”
Brokmeier, L. L., Ilic, L., Haas, S., Uter, W. J. C., Heppt, M. V., Gefeller, O., & Kaiser, I. (2025) · Healthcare · doi:10.3390/healthcare13151912 · Cited by 2
“Furthermore, the method of forward and backward citation tracking [27,28] was used.”
Yager, J. (2025) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.20240054 · Cited by 2
Trustworthy Learning Analytics for Smart Learning Ecosystems
Svetec, B., & Divjak, B. (2025) · Interaction design & architecture(s)/ID&A Interaction design & architecture(s) · doi:10.55612/s-5002-064-002sp · Cited by 4
Spatial land conflict typologies in the Atlantic Forest of South America
Anticoli, A., von Below, J., Eleutério, A. A., Pozzo, A. O., Trumper, A., & Rodríguez, M. P. (2025) · Journal of Rural Studies · doi:10.1016/j.jrurstud.2025.103847 · Cited by 10
Percival, M. A., Anderson, K. B., Pasco, J. A., Hosking, S., & Hyde, N. K. (2025) · Reproductive Female and Child Health · doi:10.1002/rfc2.70037 · Cited by 2
Koç, H., & Hynes, J. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-032-04375-7_4 · Cited by 2
Madhushanie, N., Vidanagamachchi, S., & Arachchilage, N. A. G. (2025) · Computers & Security · doi:10.1016/j.cose.2025.104659 · Cited by 4
Subayani, N. W., Rahaimah, S., & Norazilawati, N. (2025) · IJORER International Journal of Recent Educational Research · doi:10.46245/ijorer.v6i5.884 · Cited by 7
“The included articles' reference lists were manually checked for additional relevant studies (Wohlin, 2014), increasing the review's transparency, repeatability, and validity.”
IoT botnets Unveiled: architectural analysis, threat vectors, and cutting-edge detection techniques
Garg, U., Mishra, P., Gupta, N., & Pilli, E. S. (2025) · Cluster Computing · doi:10.1007/s10586-025-05633-1 · Cited by 5
“We followed the guidelines provided in existing literatures [9, 10] for search and selection strategy.”
Application of AI to formal methods — an analysis of current trends
Stock, S., Dunkelau, J., & Mashkoor, A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10729-8 · Cited by 8
“For the 89 entries, we conducted an extensive snowballing procedure (Wohlin 2014) where we conducted both forward and backward snowballing.”
van der Ven, R. G. F. M., Westra, D., van Erning, F. N., de Hingh, I. H. J. T., & Paulus, A. T. G. (2025) · Journal of Cancer Policy · doi:10.1016/j.jcpo.2025.100651 · Cited by 2
Bialek, L., Groenboom, R., & Andrikopoulos, V. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107931 · Cited by 3
Regionalising the circular economy: A three-step integrative review and gap analysis
Ruiz, A. M. C., Sanches-Pereira, A., & Zaman, A. U. (2025) · Cleaner Environmental Systems · doi:10.1016/j.cesys.2025.100371 · Cited by 11
Exploring Human-Centric Dimensions in Blockchain Smart Contracts
Ghanmi, H. A., Ahmadjee, S., Rami Bahsoon, A., & Adeyemo, H. B. (2025) · ACM Computing Surveys · doi:10.1145/3777414 · Cited by 5
“To enhance our coverage, we employed forward and backward snowballing techniques [123], mitigating the risks of missed studies by automated searches.”
Lima, B., Hasan, F., Kannan, P., Schnell, M., Mafra, A., Couffignal, S., & Backes, C. (2025) · Frontiers in Oncology · doi:10.3389/fonc.2025.1679408 · Cited by 4
Haghighat, S., & Zhang, R. (2025) · Indoor Environments · doi:10.1016/j.indenv.2025.100137 · Cited by 7
Integrating the SDGs into SMEs: A systematic review of approaches, barriers, and assessment methods
Le, H. T., Pappa, G., & Lattanzi, N. (2025) · Cleaner Production Letters · doi:10.1016/j.clpl.2025.100119 · Cited by 8
Vibe Coding: A Multivocal Systematic Mapping Study
Beaulieu, N., Dascalu, S. M., & Hand, E. (2025) · ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD) · doi:10.1109/snpd65828.2025.11254176 · Cited by 2
“F RAMING [38], [39]. publications selected, the majority reference the topic and modern adjacent concepts directly.”
InfluentialSystematic Mapping of AI‐Based Approaches for Requirements Prioritization
Limaylla-Lunarejo, M.-I., Condorí-Fernández, N., & Luaces, M. R. (2025) · IET Software · doi:10.1049/sfw2/8953863 · Cited by 3
“This mapping study was conducted following the guidelines provided by Wohlin [16] and employing a snowballing procedure in two phases.”
A Catalog of Fairness-Aware Practices in Machine Learning Engineering
Voria, G., Sellitto, G., Ferrara, C., Abate, F., Lucia, A. D., Ferrucci, F., Catolino, G., & Palomba, F. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3727967.3756824 · Cited by 18
Determining Design Challenges for Post-Disaster Temporary Housing Units Using the DEMATEL Method
Serter, M., & Ayalp, G. G. (2025) · Buildings · doi:10.3390/buildings16010033 · Cited by 2
Hernández, H., Toro, M. F., & López-Retamales, S. (2025) · Buildings · doi:10.3390/buildings16010074 · Cited by 5
A Systematic Review of Federated and Cloud Computing Approaches for Predicting Mental Health Risks
Fiaz, I., Kanwal, N., & Ahmad, A. A.-S. (2025) · Sensors · doi:10.3390/s26010229 · Cited by 8
“The automated search was complemented by a backward snowballing procedure in which the reference lists of all full-text articles were checked for additional eligible studies [24,25].”
Guzmán, E., Andrés, B., & Poler, R. (2026) · International Journal of Production Economics · doi:10.1016/j.ijpe.2026.109927 · Cited by 4
A Threat-Oriented Study of API Security Challenges in CI/CD Pipelines
Saleh, S. M., Ifat, M. N. A., Madhavji, N. H., & Steinbacher, J. (2025) · IEEE International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom67567.2025.11331540 · Cited by 3
“A backwards snowballing [17] step added additional papers, bringing the total to 56.”
Gender-transformative social protection: returning to the feminist debates
Kuss, M. K., & Cookson, T. P. (2025) · Policy and Society · doi:10.1093/polsoc/puaf048 · Cited by 3
Rehman, I. U., Vaněček, D., Chakareski, J., & Guralnick, D. A. (2026) · Virtual Reality · doi:10.1007/s10055-025-01309-8 · Cited by 3
“Moreover, manual searches and the snowballing method Wohlin ( 2014 ) were also used to discover more relevant papers.”
Steller, N., & Möllering, G. (2026) · European Management Review · doi:10.1111/emre.70041 · Cited by 4
Systematic Review of Identity-Centric Security in Cloud-Native CI/CD Pipelines
Saleh, S. M., Madhavji, N. H., & Steinbacher, J. (2025) · International Conference on Cloud Computing and Internet of Things (CCIOT) · doi:10.1145/3785520.3785525 · Cited by 2
“In parallel, backward and forward snowballing [70] identified an additional 65 records.”
Towards a Conceptual Model for Ai-Driven Web Accessibility Remediation: A Prompt-Based Approach
Vera-Amaro, G., & Rojano-Cáceres, J. R. (2025) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft66928.2025.00044 · Cited by 2
“…its full sensitivity, ensuring no QGS study was missed, with acceptable precision [29] of 4.87% as shown in Table VI. Step 6: Retrieve studies via other methods: To conclude the identification process, additional studies were retrieved through snowballing, following the guidelines proposed by [33].”
InfluentialDeng, H., Chen, H., & Dong, Y. (2026) · Journal of Educational Computing Research · doi:10.1177/07356331261424211 · Cited by 7
“Additionally, snowballing approach, was also utilized to fi nd the articles that were not extracted in database search through citation checking (Wohlin, 2014).”
Mahlaole, S. T., & Malebana, M. J. (2026) · Discover Education · doi:10.1007/s44217-026-01272-9 · Cited by 5
What to assess in circular buildings: key performance indicators for circular transformation
Mani, S., Karunasena, G., Hasan, A., & Hosseini, M. R. (2026) · Smart and Sustainable Built Environment · doi:10.1108/sasbe-11-2025-0708 · Cited by 5
Coussa, A. (2026) · Circular Economy and Sustainability · doi:10.1007/s43615-026-00836-z · Cited by 4
“The latter consists of using the reference list of papers to identify new papers to include; that can after review, be included in the articles database [49].”
Batool, H., & Batool, I. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2631667 · Cited by 3
Advancing hybrid spatial dynamic modeling for urban sustainable transitions
Zhang, H., & Cai, Z. (2026) · Discover Cities · doi:10.1007/s44327-026-00220-4 · Cited by 3
Resilience capability for the circular economy: a conceptual framework
Santos, J. E. A., & Gohr, C. F. (2026) · Environment Systems & Decisions · doi:10.1007/s10669-026-10079-w · Cited by 8
“0 0 1 Beske P, Seuring S (2014) Putting sustainability into supply chain management.”
Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review
Zhang, J., Hu, X., Gao, C., Xia, X., & Li, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3802827 · Cited by 13
Rivero-Romero, O., Unfried-Silgado, J., & Colorado, H. A. (2026) · Journal of Materials Research and Technology · doi:10.1016/j.jmrt.2026.03.240 · Cited by 6
Zaland, S., Saeed, N., Aziz, T., Tao, Z., & Kang, W. (2026) · Construction and Building Materials · doi:10.1016/j.conbuildmat.2026.146141 · Cited by 2
Zarea, H. (2026) · Business Strategy and the Environment · doi:10.1002/bse.70788 · Cited by 5
Artificial intelligence in washing machines: a systematic review
Pembe, F. C., Bilgen, İ., Delibalta, M., Kiremit, E., & Haklıdır, M. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12027-w · Cited by 2
The effects of educational robotics on students’ learning outcomes: A three-level meta-analysis
Lu, Y., & Geng, S. (2026) · Educational Research Review · doi:10.1016/j.edurev.2026.100790 · Cited by 5
Radar-Based Human-Computer Interaction When Sensing Through Materials
Attygalle, N. T., Kljun, M., Sluÿters, A., et al. (2026) · Human-computer interaction series · doi:10.1007/978-3-032-04327-6_4 · Cited by 2
Conflict-Based Models for Real-Time Crash Risk Assessment: A State-of-the-Art Review
Jackai, I. N., Feudjio, S. L. T., Lordswill, N. T., et al. (2026) · Future Transportation · doi:10.3390/futuretransp6030107 · Cited by 2
Shah, H., Verma, S., & Srivastava, V. (2026) · Journal of Promotion Management · doi:10.1080/10496491.2026.2672505 · Cited by 4
“The forward and backward citation tracking (snowballing) was also used to retrieve more relevant studies missed by the initial query (Wohlin, 2014).”
Organizational resilience: a promising concept for organizational strategy or just a new buzzword?
Oliveira, L. M. D., Barbosa, A. P. P. L., & Marx, R. (2026) · Multinational Business Review · doi:10.1108/mbr-08-2024-0158 · Cited by 4
Schulte, N., & Kanbach, D. K. (2026) · Management Review Quarterly · doi:10.1007/s11301-026-00611-2 · Cited by 21
Large language models in model-driven engineering: a systematic mapping study
Zhang, W., Jiang, B., Yuhong, F., Cheng, H., Hummel, M., Scotti, V., Hagel, N., Li, J., Großmann, G., Stumptner, M., et al. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10921-4 · Cited by 6
“To reduce threats to construct validity and cover as many relevant studies as possible, we adopted a snowballing approach to supplement the automatic search described above [113].”
GROUPROFILE: Um Modelo de Gerenciamento de Perfis de Grupos de Aprendizes
Sparremberger, A. S., Barbosa, J. L. V., & Gluz, J. C. (2017) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/cbie.sbie.2017.1687 · Cited by 2
A Domains Oriented Framework of Recent Machine Learning Applications in Mobile Mental Health
Theilig, M.-M., Blankenhagel, K. J., & Zarnekow, R. (2018) · Lecture notes in information systems and organisation · doi:10.1007/978-3-030-01087-4_20 · Cited by 3
A Systematic Mapping Review of Memory Leak Detection Techniques
de Sena, G. O., & Matias, R. (2018) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2018.00017 · Cited by 2
“Next, we analyzed all papers listed in the reference section of the selected 71 studies, and added to the work sample the most cited references as part of the snowballing process [9] adopted in this work.”
MethodA Requirements Definition Framework for the Robotic Systems Domain - An Exploratory Study
Albuquerque, D. W., Castro, J. B., & Sousa, A. (2018) · doi:10.17771/pucrio.wer.inf2018-49 · Cited by 2
Analysis of Task Management in Virtual Academic Teams
Nagel, S., & Schwade, F. (2020) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2020.057 · Cited by 2
“An additional exhaustive literature review following [4, 5, 48] established the current state of research on TM.”
BackgroundWicaksono, A., Sodri, A., & Chairani, E. (2020) · E3S Web of Conferences · doi:10.1051/e3sconf/202021103003 · Cited by 6
Software Process Improvement Diagnostic: A Snowballing Systematic Literature Review
Ecar, M., da Silva, J. P. S., Amorim, N. S., Rodrigues, E. M., Basso, F. P., & Solda, T. G. (2020) · Latin American Computer Conference (CLEI) · doi:10.1109/clei52000.2020.00025 · Cited by 4
Towards an Anatomy of Software Craftsmanship
Sundelin, A., González‐Huerta, J., Wnuk, K., & Gorschek, T. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3468504 · Cited by 7
Spennemann, D. H. R. (2021) · Sports · doi:10.3390/sports9120172 · Cited by 4
Audiojogos educacionais: um Mapeamento Sistemático da Literatura
Carvalho, W. R. B., Rodriguez, C. L., & da Rocha, R. V. (2021) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2021.218377 · Cited by 2
“O método de snowballing para frente é definido como a análise das referências e citações dos estudos primários selecionados, enquanto o para trás refere-se aos trabalhos que citam os estudos primários (Wohlin, 2014), o que possibilita a adição de trabalhos relevantes.”
BackgroundAdaptation of Modern Agile Practices in Global Software Engineering
Mansoor, M., Khan, M. W., Rizvi, S. S. H., Hashmani, M. A., & Zubair, M. (2021) · IGI Global eBooks · doi:10.4018/978-1-6684-3702-5.ch013 · Cited by 2
Could jamstack be the future of web applications architecture? an empirical study
Marković, D., Scekic, M., Bucaioni, A., & Cicchetti, A. (2022) · ACM Symposium on Applied Computing · doi:10.1145/3477314.3506991 · Cited by 10
“We performed a closed recursive backward and forward snowballing activity [25], but we did not find additional relevant studies.”
MethodSecurity and Privacy for Blockchain: A Systematic Mapping Study
Yang, J., Bi, H., Dai, F., Liang, Z., & Qiang, Z. (2021) · IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) · doi:10.1109/dasc-picom-cbdcom-cyberscitech52372.2021.00081 · Cited by 2
Meninas negras em programas para educação de superdotados. Uma revisão de literatura
Gomide, F. B., & García, A. I. (2022) · Revista Iberoamericana de Educación · doi:10.35362/rie8814842 · Cited by 9
“Realizamos os procedimentos descritos por Wohlin (2014) que significou usar a lista de referências bibliográficas de cada um dos 11 artigos para identificar novos artigos.”
Background
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.