Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 1,701–1,800 of 3,934
Gregory, P., Lassenius, C., Wang, X., & Kruchten, P. B. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-78098-2 · Cited by 25
Investigating Digital Twin: A Systematic Mapping Study
Guzina, L., Ferko, E., & Bucaioni, A. (2022) · Advances in transdisciplinary engineering · doi:10.3233/atde220164 · Cited by 12
A Flying Reign of Terror? The Who, Where, When, What, and How of Non-state Actors and Armed Drones
Haugstvedt, H. (2023) · Journal of Human Security · doi:10.12924/johs2023.19010001 · Cited by 16
Grounded Architectures: Using Grounded Theory for the Design of Software Architectures
Gidey, H. K., Marmsoler, D., & Eckhardt, J. (2017) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsaw.2017.41 · Cited by 15
“A combination of automated search in databases and snowballing [37] is used to get better results.”
MethodModel-Based Mutation Operators for Timed Systems: A Taxonomy and Research Agenda
Ortiz, J., Perrouin, G., Amrani, M., & Schobbens, P. (2018) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2018.00045 · Cited by 12
“Similarly, we did not perform a rigourous snowballing as described by Wohlin [20]: we only performed a backward snowballing starting from all the papers cited in this survey, and marginally operated forward snowballing on the main papers listed in Table I, using the same exclusion criteria as the ones used for the sea…”
MethodAdie, L. E., Harris, L. R., & Wyatt‐Smith, C. (2020) · Teaching and Teacher Education · doi:10.1016/j.tate.2019.103012 · Cited by 18
Revisiting Legacy Software System Modernization
Khadka, R. (2016) · Utrecht University Repository (Utrecht University) · Cited by 45
Prediction of COVID-19 using Machine Learning Techniques
Matta, D. M., & Saraf, M. K. (2020) · DiVA (Blekinge Institute of Technology) · Cited by 20
Initiatives and Challenges of Using Gamification in Software Engineering: A Systematic MappingPreprint
Porto, D. D. P., Jesus, G., Ferrari, F. C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07115 · Cited by 11
Reframing safety barriers as socio-technical systems: a review of the hydrogen sector
Subedi, A., Bucelli, M., & Paltrinieri, N. (2025) · Safety Science · doi:10.1016/j.ssci.2025.106995 · Cited by 11
Federated Learning for Smart Cities: A Thematic Review of Challenges and Approaches
Alterkawi, L., & Dib, F. K. (2025) · Future Internet · doi:10.3390/fi17120545 · Cited by 23
“To strengthen coverage in underrepresented domains such as EV infrastructure, disaster resilience, and grid optimization, we also conducted backward and forward citation tracking from foundational works [28].”
Artificial intelligence in K-12 education: An umbrella review
Huang, R., Yin, Y., Zhou, N., & Lang, F. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100519 · Cited by 37
Towards Resilient and Sustainable Production Systems: A Research Agenda
Säfsten, K., Harlin, U., Johansen, K., et al. (2022) · Advances in transdisciplinary engineering · doi:10.3233/atde220195 · Cited by 10
Automotive Service-oriented Architectures: a Systematic Mapping Study
Kukulicic, N., Samardzic, D., Bucaioni, A., & Mubeen, S. (2022) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa56994.2022.00077 · Cited by 13
“We complemented the initial automatic search with an exhaustive backward snowballing as suggested by Wohlin et al. [11].”
MethodExamining the commonalities and knowledge gaps in coastal zone vegetation simulation models
Piercy, C. D., Charbonneau, B. R., Russ, E. R., & Swannack, T. M. (2023) · Earth Surface Processes and Landforms · doi:10.1002/esp.5565 · Cited by 15
“A literature review was conducted winter and spring 2021 using a combination of backward and forward snowballing utilizing start sets derived from an initial search using the term “ model ” plus the habitat name as well as any well-known models of the habitat (Wohlin, 2014).”
MethodSpennemann, D. H. R., Parker, M., & Bond, J. (2023) · Environments · doi:10.3390/environments10090163 · Cited by 10
“During the full text-screening and evaluation process, an additional 59 papers (mainly on building effects on pollutant dispersal) were identified via snowballing (i.e., iteratively checking references in identified papers) [24] and added to the analysis.”
Cultural change in servitization – a conceptual review and framework
Biesinger, B., Hadwich, K., & Bruhn, M. (2024) · Journal of Service Theory and Practice · doi:10.1108/jstp-03-2023-0074 · Cited by 16
Organizational Changes in Adopting Agile Approaches: A Systematic Literature Review
Perides, M. P. N., & Vasconcellos, L. (2024) · The Journal of Applied Behavioral Science · doi:10.1177/00218863241229704 · Cited by 23
“The references cited by these studies were then analyzed to identify potential papers that could contribute to this research, but which had not been identi fi ed in the initial selection (Webster & Watson, 2002; Wohlin, 2014).”
Towards Smarter Security Orchestration and Automatic Response for CPS and IoT
Nguyen, P. H., Dautov, R., Song, H., Rego, A., Iturbe, E., Rios, E., Sagasti, D., Nicolas-Barreales, G., Valdés, V., Mallouli, W., et al. (2023) · IEEE International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom59040.2023.00055 · Cited by 11
“In this section, we discuss the primary SOAR studies that we have found using the snowballing method [21].”
BackgroundSelfish mining attack in blockchain: a systematic literature review
Madhushanie, N., Vidanagamachchi, S., & Arachchilage, N. A. G. (2024) · International Journal of Information Security · doi:10.1007/s10207-024-00849-5 · Cited by 15
“The final search (Phase 3) employed the backward snowballing method [ 22], which entails identifying further papers based on references in previously retrieved publications and their citations.”
Tehan, P. E., Perrin, B., Lazzarini, P. A., Al-Busaidi, I. S., & Carroll, M. (2024) · Journal of Foot and Ankle Research · doi:10.1002/jfa2.12012 · Cited by 11
“Additional publications were identified by backward snowballing checks of reference lists [27].”
BackgroundStudy of Code Smells: A Review and Research Agenda
Tandon, S., Kumar, V. S., & Singh, V. (2024) · International Journal of Mathematical Engineering and Management Sciences · doi:10.33889/ijmems.2024.9.3.025 · Cited by 11
Cavalcante, E., Batista, T., & Oquendo, F. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2697 · Cited by 11
“The snowballing technique, 56 which uses the reference list of a study (backward snowballing) or citing references to it (forward snowballing) to identify additional studies, is often used to increase literature coverage in secondary studies, thus contributing to minimizing threats to study…”
An Empirically Grounded Reference Architecture for Software Supply Chain Metadata Management
Tran, N. K., Pallewatta, S., & Babar, M. A. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661212 · Cited by 14
“This preliminary selection was expanded using the snowballing sampling technique [35].”
MethodModular Monolith: Is This the Trend in Software Architecture?
Su, R., & Li, X. (2024) · doi:10.1145/3643657.3643911 · Cited by 10
“We also adopted the "snowballing" method in the process to extract more relevant information [12].”
MethodAugmented Reality on the Move: A Systematic Literature Review for Vulnerable Road Users
Stefanidi, H., Tatzgern, M., & Meschtscherjakov, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3676490 · Cited by 18
“After this exclusion round, we scanned the reference lists of these papers and another 12 were added, following Wohlin’s guidelines for snowballing in systematic reviews [120].”
Al-Qazzaz, I., Osorio-Sandoval, C. A., Tokbolat, S., & Thermou, G. E. (2024) · Built Environment Project and Asset Management · doi:10.1108/bepam-12-2023-0229 · Cited by 15
Analysis of Influencing Factors of Level 3 Automated Vehicle Takeover: A Literature Review
Guo, H., Qiu, H., Zhou, Y., & Deng, Y. (2024) · Sustainability · doi:10.3390/su16198345 · Cited by 14
The Skill Gap in Software Industry: A Mapping Study
Diniz, W., Valença, M., França, C., Santos, A. D. O., & Pincovsky, M. (2024) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2024.3351 · Cited by 20
Balancing stakeholder engagement in climate action: A symbiotic typology approach
Badu, J., Kruke, B. I., & Sætren, G. B. (2024) · Environmental Science & Policy · doi:10.1016/j.envsci.2024.103953 · Cited by 13
Alshammari, S. S., Soh, B., & Li, A. (2025) · Information · doi:10.3390/info16020153 · Cited by 23
Climatic Influence on Growth Performance of Abies spectabilis in the Himalayas
Pandey, K. P., Wellstein, C., Bräuning, A., & Bhuju, D. R. (2025) · Forests · doi:10.3390/f16030473 · Cited by 10
Marketing in the anthropocene: A future agenda for research and practice
Bocken, N. M. P., Nießen, L., Gossen, M., Das, A., & Zielińska, M. (2025) · AMS Review · doi:10.1007/s13162-025-00300-5 · Cited by 22
“…the supplementary search strategies of backward snowballing (i.e., screening reference lists of included studies) and forward snowballing (i.e., screening articles that cite included studies) as well as expert consultations to uncover more relevant articles (Baldassarre et al., 2020; Wohlin, 2014).”
Digital Technologies: Description, Classification, and Links to Circular Economy
Piedra‐Muñoz, L., García‐Granero, E. M., Tarpani, R. R. Z., & Gallego‐Schmid, A. (2025) · Business Strategy and the Environment · doi:10.1002/bse.4312 · Cited by 21
Citybabu, G., & Yamini, S. (2025) · International Journal of Productivity and Performance Management · doi:10.1108/ijppm-05-2024-0295 · Cited by 12
Comparison of static analysis architecture recovery tools for microservice applications
Schneider, S., Bakhtin, A., Li, X., Soldani, J., Antonio Brogi, A., Cerny, T., Scandariato, R., & Taibi, D. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10686-2 · Cited by 23
“Out of topic - relevant terms are used in a different context – Source describes different aspects of microservice recovery – Source describes closely related tasks such as monolith to”
Is there a Future for Empirical Software Engineering?
Wohlin, C. (2016) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962641 · Cited by 9
“[2], [3], [6] and [7]) for conducting systematic literature studies have been formulated and they should be carefully followed.”
BackgroundSelf-citationGuidelines for utilizing change impact analysis when estimating effort in agile software development
Tanveer, B. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084284 · Cited by 13
“Similarly, we identified systematic literature studies [19, 11] and surveys of practice [18, 3] done in the context of effort estimation in agile development and used them as start set for performing forward snowballing [20] for answering RQ3.”
MethodKalantaridis, C., & Küttim, M. (2020) · The Journal of Technology Transfer · doi:10.1007/s10961-019-09757-6 · Cited by 14
“This was followed by snowballing that meant using the reference list of selected papers or the citations to identify additional papers (Wohlin 2014).”
MethodDesigning Mobile Training Content: Challenges and Open Issues
Al-Amri, S., Noor, N. F. B. M., Hamid, S. B., et al. (2020) · IEEE Access · doi:10.1109/access.2020.3006712 · Cited by 10
“A secondary method of searching called a snowballing technique was also used, which involved looking up the references of references and bibliographies by using key articles or books related to the research topic [31].”
MethodUnderstanding practitioners’ strategies to handle test smells: a multi-method study
Santana, R., Fernandes, D. D., Campos, D., et al. (2021) · Brazilian Symposium on Software Engineering · doi:10.1145/3474624.3474639 · Cited by 10
A mapping study on mutation testing for mobile applications
da Silva, H. N., Lima, J. P., Vergílio, S. R., & Endo, A. T. (2021) · Software Testing Verification and Reliability · doi:10.1002/stvr.1801 · Cited by 12
“Next, we performed a snowballing reading (forward and backward snowballing procedures, following Wohlin ’ s guidelines [30]), using the control group as a starting set.”
MethodPlatelet rich plasma effectiveness in bowel anastomoses: A systematic review
Geropoulos, G., Psarras, K. K., Giannis, D., et al. (2021) · World Journal of Gastrointestinal Surgery · doi:10.4240/wjgs.v13.i12.1736 · Cited by 10
Tasoudis, P., Varvoglis, D. N., Vitkos, E. N., et al. (2022) · Perfusion · doi:10.1177/02676591221095468 · Cited by 10
Vaccine Innovation Meta-Model for Pandemic Contexts
Fialho, B. D. C., Gauss, L., Soares, P. F., Medeiros, M. Z., & Lacerda, D. P. (2023) · Journal of Pharmaceutical Innovation · doi:10.1007/s12247-023-09708-7 · Cited by 14
“The snowballing was conducted backward and forward, as reasoned by Wohlin [35].”
Reproducible Searches in Systematic Reviews: An Evaluation and Guidelines
Li, Z., & Rainer, A. (2023) · IEEE Access · doi:10.1109/access.2023.3299211 · Cited by 10
“Essentially, we snowballed [19] from tertiary studies.”
BackgroundInfluentialAssessment of antecedents of financial literacy among young adults: hybrid BWM and ISM approach
Saharan, A., Saharan, A., Pandey, K. K., & Rao, T. J. (2023) · International Journal of Social Economics · doi:10.1108/ijse-11-2022-0746 · Cited by 13
Klemmer, J. H., Gutfleisch, M., Stransky, C., Acar, Y., Sasse, M. A., & Fahl, S. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3623072 · Cited by 27
Bots in Software Development: A Systematic Literature Review and Thematic Analysis
Moguel-Sánchez, R., Palacios, C. S. M., Ocharán-Hernández, J. O., Limón, X., & Sánchez-García, Á. J. (2023) · Programming and Computer Software · doi:10.1134/s0361768823080145 · Cited by 13
“To expand upon the studies selected from the automatic search, a snowballing search for SE was executed based on the guidelines for snowballing in systematic literature studies and replication in software engineering [21].”
Predicting luxury purchases: A new comprehensive framework and research roadmap
Al‐Issa, N., Dens, N., Moons, I., & Ali, O. (2024) · Cogent Business & Management · doi:10.1080/23311975.2023.2285815 · Cited by 14
“In addition, a backward snowballing approach using a reference rundown (Wohlin, 2014) was used to find articles that were missed by the process.”
Lunn, S. J., Bell-Huff, C. L., & LeDoux, J. (2024) · Collaborative Network for Engineering and Computing Diversity Conference (CoNECD) · doi:10.18260/1-2--39112 · Cited by 14
“Snowballing was also performed to maximize relevant sources [47].”
BackgroundTools Reconstructing Microservice Architecture: A Systematic Mapping Study
Bakhtin, A., Li, X., Soldani, J., Brogi, A., Cerny, T., & Taibi, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66326-0_1 · Cited by 14
Interoperability in Systems-of-Information Systems: A Systematic Mapping Study
Fernandes, J. C., Neto, V. V. G., & dos Santos, R. P. (2018) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3275245.3275259 · Cited by 19
“…1º filtro: leitura de títulos, abstracts e palavras-chave; (4) 2º filtro: leitura da introdução e conclusão; (5) aplicação do backward snowballing [2] (com filtragem abrangendo etapas semelhantes às apresentadas na seleção dos trabalhos retornados na busca automatizada); (6) 3º filtro: leitura…”
BackgroundSoftware Deployment on Heterogeneous Platforms: A Systematic Mapping Study
Andrade, H., Schroeder, J., & Crnković, I. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2932665 · Cited by 15
Can You Repeat That? Exploring the Definition of a Successful Model Replication in Health Economics
McManus, E., Turner, D., & Sach, T. (2019) · PharmacoEconomics · doi:10.1007/s40273-019-00836-y · Cited by 13
“Instead, a simple search strategy was designed with the intention of identifying a subset of the relevant literature, which could act as a springboard to identify further relevant studies through the use of snowballing and citation tracking [39, 40].”
MethodMigration to Microservices: Barriers and Solutions
Ghofrani, J., & Bozorgmehr, A. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-32475-9_20 · Cited by 10
A SYSTEMATIC MAPPING STUDY ON CLOUD-BASED MOBILE APPLICATION TESTING
Ya’u, B. I., Salleh, N., Nordin, A., Idris, N. B., Abas, H., & Alwan, A. A. (2019) · Journal of Information and Communication Technology · doi:10.32890/jict2019.18.4.5 · Cited by 14
“Hence, a searching technique known as, backward snowballing, was applied to examine the reference list of the selected primary studies (Wohlin, 2014).”
MethodSeppänen, P., Oivo, M., & Liukkunen, K. (2016) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice/itmc39735.2016.9026055 · Cited by 16
“Mapping was expanded by a snowballing round on the first-level references of the included publications following the guidelines set out in [16].”
BackgroundReserve Military Service: A Social Constructionist Perspective
Griffith, J. E., & Ben‐Ari, E. (2020) · Armed Forces & Society · doi:10.1177/0095327x20917165 · Cited by 14
“The “snowballing” technique (Wohlin, 2014) was used, where references and citations of already identified articles were examined for inclusion relative to the study’s aims.”
MethodNegative Effects of Design-Bid-Build Procurement on Construction Projects
Kortenko, S., Koskela, L., Tzortzopoulos, P., & Haghsheno, S. (2020) · Annual Conference of the International Group for Lean Construction · doi:10.24928/2020/0141 · Cited by 11
Passing the Data Baton: A Retrospective Analysis on Data Science Work and Workers
Crisan, A., Fiore-Gartland, B., & Tory, M. K. (2020) · IEEE Visualization in Data Science (VDS) · doi:10.1109/vds51726.2020.00005 · Cited by 14
“We used a “snowball approach” [85] to conduct our literature review.”
MethodA meta-analysis of the effects of tree retention on shrubland birds
Akresh, M. E., King, D. I., Lott, C. A., Larkin, J. L., & D’Amato, A. W. (2020) · Forest Ecology and Management · doi:10.1016/j.foreco.2020.118730 · Cited by 14
Vieira, M., Hauck, J. C. R., & Matalonga, S. (2020) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3439961.3439976 · Cited by 14
“After the third iteration the backward snowballing technique [52] was manually applied to select studies from selected articles references.”
MethodInfluentialAyele, W., & Juell-Skielse, G. (2022) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-030-73103-8_53 · Cited by 13
“Additionally, to ensure that as many relevant articles as possible are selected, the snowballing technique by [45] is followed.”
MethodSTEAM – Arts Integration Frameworks for Transdisciplinarity
Rodier, C., Galaleldin, M., Boudreau, J., Anis, H., & Peyton, L. (2021) · Proceedings of the Canadian Engineering Education Association (CEEA) · doi:10.24908/pceea.vi0.14918 · Cited by 13
““Snowballing refers to using the reference list of a paper or the citations to the paper to identify additional papers”[8].”
BackgroundA Systematic Mapping Study on Approaches for Al-Supported Security Risk Assessment
Erdogan, G., Garcia-Ceja, E., Hugo, Å., Nguyen, P. H., & Sen, S. (2021) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac51774.2021.00107 · Cited by 10
“We also performed backward snowballing [8] by manually checking the references of the 33 selected papers to find other primary studies.”
MethodSystematic mapping of the literature on mobile apps for people with Autistic Spectrum Disorder
Krause, M., & Neto, M. A. C. (2021) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3470482.3479616 · Cited by 10
“1Snowballing se refere ao uso da lista das referências de um artigo ou as citações do artigo para identificar artigos adicionais [37].”
BackgroundLife Cycle Assessment of Network Traffic: A Review of Challenges and Possible Solutions
Billstein, T., Björklund, A., & Rydberg, T. (2021) · Sustainability · doi:10.3390/su132011155 · Cited by 17
“Iterative snowballing was performed to complement the initial search with further relevant literature [21], with the purpose of identifying literature that mention challenges linked to measuring the life cycle impact of network traffic.”
MethodCross-Language Plagiarism Detection: Methods, Tools, and Challenges: A Systematic Review
Botto-Tobar, M., Serebrenik, A., & Van den Brand, M. G. J. (2022) · International Journal on Advanced Science, Engineering and Information Technology · doi:10.18517/ijaseit.12.2.14711 · Cited by 10
“Further, the search-based approach is sensitive to the choice of the keywords, so to compensate for this, we have performed the snowballing method [11].”
BackgroundMethodRain Intrusion through Horizontal Joints in Façade Panel Systems—Experimental Investigation
Støver, E. A., Sundsøy, M. H., Andenæs, E., et al. (2022) · Buildings · doi:10.3390/buildings12101497 · Cited by 10
“This method is called snowballing or citation chaining [43].”
MethodA Survey on Data-Driven Evaluation of Competencies and Capabilities Across Multimedia Environments.
Strukova, S., Ruipérez‐Valiente, J. A., & Mármol, F. G. (2023) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2022.10.004 · Cited by 11
Oluoch, W. A., Whitney, C., Termote, C., Borgemeister, C., & Schmitt, C. B. (2023) · Journal of Ethnobiology and Ethnomedicine · doi:10.1186/s13002-023-00584-6 · Cited by 12
“from the literature We extracted threats and management options for WEPs from published literature using a snow-ball sampling approach [51].”
MethodThe Impact of Internet of Things on Software Business Models
Wnuk, K., & Murari, B. T. (2016) · Lecture notes in business information processing · doi:10.1007/978-3-319-40515-5_7 · Cited by 12
Ginny, Kumar, C., & Naik, K. (2020) · The Journal of Supercomputing · doi:10.1007/s11227-020-03312-z · Cited by 18
Comparison of Agile Maturity Models: Reflecting the Real Needs
Tuncel, D., Korner, C., & Plösch, R. (2020) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa51224.2020.00019 · Cited by 13
“Snowballing procedure was conducted in an alignment with [26], to make sure no relevant study has been overlooked prior to the refinement step.”
MethodThe Use of Immersive Virtual Reality for Vocabulary Acquisition: a Systematic Literature Review
Palmeira, E. G. Q., Martin, V. B. S., Gonçalves, V. B., Moraes, Í. A., Júnior, E. A. L., & Cardoso, A. (2020) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/cbie.sbie.2020.532 · Cited by 15
“1In this process, the list of references of the selected studies is analyzed in an attempt to identify additional relevant studies that were not found by applying the search string [Wohlin 2014] IX Congresso Brasileiro de Informática na Educação (CBIE 2020) Anais do XXXI Simpósio Brasileiro de Informática na Educação…”
MethodInvestigating Factors That Influence Developers' Experience in Mobile Software Ecosystems
Fontão, A., Dias‐Neto, A. C., & Viana, D. (2017) · IEEE/ACM Joint International Workshop on Software Engineering for Systems-of-Systems and Workshop on Distributed Software Development, Software Ecosystems and Systems-of-Systems (SESoS/WDES) · doi:10.1109/jsos.2017.10 · Cited by 15
“After defining the start set, the Forward Snowballing (identifying new papers based on those papers citing the paper being examined) procedure was implemented.”
BackgroundMethodInfluentialAI in Automated and Remote UX Evaluation: A Systematic Review (2014–2024)
Liu, J. (2025) · Advances in Human-Computer Interaction · doi:10.1155/ahci/7442179 · Cited by 22
“In contrast, AI-enhanced automation employs ML and predictive modeling to simulate complex user behaviors, adapt to changing interfaces, and identify subtle usability problems in real time [22, 23].”
Tendencies in Multi-Agent Systems: A Systematic Literature Review
Falco, M., & Robiolo, G. (2020) · CLEI electronic journal · doi:10.19153/cleiej.23.1.1 · Cited by 10
“The present SLR followed the guidelines provided by Kitchenham [12], and complemented through the search approach known as snowballing defined by Wohlin [13].”
BackgroundMethodInfluentialDevelopment of a Token Design Framework
Schubert, N., Obermeier, D., Kohlbrenner, F., & Sandner, P. (2021) · IEEE International Conference on Decentralized Applications and Infrastructures (DAPPS) · doi:10.1109/dapps52256.2021.00009 · Cited by 9
Jahangır, R., Bulut, M., & Dinç, Y. (2022) · Property Management · doi:10.1108/pm-03-2022-0025 · Cited by 10
A systematic mapping study on machine learning methodologies for requirements management
Xu, C. Y., Li, Y., Wang, B., & Dong, S. (2022) · IET Software · doi:10.1049/sfw2.12082 · Cited by 14
Song, W., Liu, J., Tu, G.-W., Pan, L., Hong, Y., Qin, L., Wei, L., & Chen, J. (2023) · BMC Cardiovascular Disorders · doi:10.1186/s12872-023-03517-z · Cited by 14
“References from selected literatures were also manually scrutinized for potentially relevant citations using the snowball methodology [14].”
MethodShaliyar, M., & Mustafa, K. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17559-0 · Cited by 13
What About the Data? A Mapping Study on Data Engineering for AI Systems
Heck, P. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644954 · Cited by 14
Wang, X., Zhang, T., Tan, Y., Shang, W., & Li, Y. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112040 · Cited by 19
Feng, Z., Kimura, K., Trinkenreich, B., Sarma, A., & Steinmacher, I. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2024.107470 · Cited by 20
Wet markets in Southeast Asia and access to healthy diets
Hofman, M., & Trevenen-Jones, A. (2024) · Frontiers in Sustainability · doi:10.3389/frsus.2024.1320959 · Cited by 15
Woschank, M., Dallasega, P., König, A., & Hoffelner, M. (2024) · Flexible Services and Manufacturing Journal · doi:10.1007/s10696-024-09550-0 · Cited by 21
“Moreover, according to Wohlin (Wohlin 2014) we used a backward and forward snowballing approach to include relevant works.”
Pivotto, I. D., Matias, V., & Ferreira, W. D. P. (2024) · Research in Autism · doi:10.1016/j.rasd.2024.102453 · Cited by 17
Motor Imagery Signal Classification Using Adversarial Learning: A Systematic Literature Review
Mishra, S., Mahmudi, O., & Jalali, A. (2024) · IEEE Access · doi:10.1109/access.2024.3421569 · Cited by 9
“To overcome these issues, [56], [58] recommend snowballing as an alternative method to identify relevant research.”
BackgroundMethodModelling the quantification of requirements technical debt
Perera, J., Tempero, E. D., Tu, Y., & Blincoe, K. (2024) · Requirements Engineering · doi:10.1007/s00766-024-00424-3 · Cited by 11
“Backward reference snowballing [22] was performed on the final set of articles to complement the search query and to avoid the possibility of missing relevant articles.”
Values‐based barriers and good practices in sustainability‐oriented innovation management
Ivanov, K. (2024) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.3016 · Cited by 16
State of the Art of the Security of Code Generated by LLMs: A Systematic Literature Review
Ramírez, L. C., Limón, X., Sánchez-García, Á. J., & Pérez-Arriaga, J. C. (2024) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft63288.2024.00050 · Cited by 30
A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing Techniques
Elgendy, I. T., Hierons, R. M., & McMinn, P. (2025) · Software Testing Verification and Reliability · doi:10.1002/stvr.1914 · Cited by 12
Pasture monitoring using remote sensing and machine learning: A review of methods and applications
Shahi, T. B., Balasubramaniam, T., Sabir, K. S., & Nayak, R. (2025) · Remote Sensing Applications Society and Environment · doi:10.1016/j.rsase.2025.101459 · Cited by 27
A Pedagogical Framework for Developing Abstraction Skills
Begum, M., Crossley, J., Strömbäck, F., Akrida, E. C., Alpizar-Chacón, I., Evans, A. C., Gross, J. B., Haglund, P., Lonati, V., Satyavolu, C. J., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709613 · Cited by 15
“Members took responsibility for expanding this body of literature through a snowballing approach [95], where abstracts of newly identified papers were reviewed based on whether they explored abstraction theoretically, empirically, or both.”
Smart Lighting Systems: State-of-the-Art in the Adoption of the EdgeML Computing Paradigm
Paolone, G., Paesani, R., Pilotti, F., Camplone, J., Piazza, A., & Felice, P. D. (2025) · Future Internet · doi:10.3390/fi17020090 · Cited by 14
Social sustainability in manufacturing: where are we?
Blandino, G., & Montagna, F. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2464904 · Cited by 17
“Snowballing (Wohlin 2014) was applied at the beginning to explore the state of the art of the literature and to select the most relevant contributions for the objectives of the research.”
Oumoussa, I., & Saidi, R. (2025) · Computers · doi:10.3390/computers14040133 · Cited by 23
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.