Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 1,201–1,300 of 3,934
Stir Casting Routes for Processing Metal Matrix Syntactic Foams: A Scoping Review
de la Muela, A. M. S., Duarte, J., Baptista, J. S., et al. (2022) · Processes · doi:10.3390/pr10030478 · Cited by 20
Model Transformation Testing and Debugging: A Survey
Troya, J., Segura, S., Burgueño, L., & Wimmer, M. (2022) · ACM Computing Surveys · doi:10.1145/3523056 · Cited by 23
Compliance checking of software processes: A systematic literature review
Ardila, J. P. C., Gallina, B., & Muram, F. U. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2440 · Cited by 45
“After, we perform a snowballing, 70 which is a technique that aims at reaching more relevant primary studies.”
MethodA parallel programming assessment for stream processing applications on multi-core systems
Andrade, G., Griebler, D., dos Santos, R. P., & Fernandes, L. G. (2022) · Computer Standards & Interfaces · doi:10.1016/j.csi.2022.103691 · Cited by 20
A Mixed Review of Cash Flow Modeling: Potential of Blockchain for Modular Construction
Assaf, M., Hussein, M., Alsulami, B. T., & Zayed, T. M. (2022) · Buildings · doi:10.3390/buildings12122054 · Cited by 33
Braig, M., & Zeiler, P. (2022) · IEEE Access · doi:10.1109/access.2022.3233220 · Cited by 25
“Forward and backward snowballing [35] are used to supplement the searches.”
MethodAnane, W., Iordanova, I., & Ouellet‐Plamondon, C. (2023) · Digital Manufacturing Technology · doi:10.37256/dmt.3120231856 · Cited by 28
“This technique uses ‘‘forward snowballing’’ to identify where the studied article was cited and ‘‘backward snowballing’’ to see on which citations this article was based [25-27].”
MethodThe applications of machine learning in computational thinking assessments: a scoping review
Tan, B., Jin, H.-Y., & Cutumisu, M. (2023) · Computer Science Education · doi:10.1080/08993408.2023.2245687 · Cited by 29
Applying functional near-infrared spectroscopy (fNIRS) in educational research: a systematic review
Zhan, Z., Yang, Q., Luo, L., & Zhang, X. (2023) · Current Psychology · doi:10.1007/s12144-023-05094-y · Cited by 25
“Based on the starting set, we adopted the snow - balling search approach (Wohlin, 2014) by going backward to look at the reference list and going forward to look at the citations of the papers to identify additional papers for our literature review.”
A Systematic Mapping Study on Gamification Applications for Undergraduate Cybersecurity Education
Weitl-Harms, S., Spanier, A., Hastings, J. D., & Rokusek, M. (2023) · Journal of Cybersecurity Education Research and Practice · doi:10.32727/8.2023.12 · Cited by 24
Characterisation of Challenges for Testing of Adaptive Systems
Siqueira, B. R., Ferrari, F. C., Serikawa, M. A., et al. (2016) · doi:10.1145/2993288.2993294 · Cited by 20
“In short, it consists in looking for relevant studies, which are referenced by authors of primary studies that are evaluated during the search process [40].”
BackgroundExploring Knowledge Loss in Open Source Software (OSS) Projects
Rashid, M., Clarke, P. M., & O’Connor, R. V. (2017) · Communications in computer and information science · doi:10.1007/978-3-319-67383-7_35 · Cited by 29
“Backward snowballing refers to the process of extending the literature review on the basis of following the trail offered up by the initial set of paper identified [20].”
BackgroundSalam, M. A., & Khan, S. U. (2018) · Journal of Software Evolution and Process · doi:10.1002/smr.1939 · Cited by 26
“To identify more papers, we performed a snowballing technique 75 by looking through the references of the final SLR ‐ selected papers and searching by author, using author names of the finally selected papers.”
MethodA Systematic Review of Cost Reduction Techniques for Mutation Testing: Preliminary Results
Ferrari, F. C., Pizzoleto, A. V., & Offutt, J. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00021 · Cited by 20
“To mitigate this threat, we used Wohlin’s alternative suggestion and contacted authors of primary studies to ask for additional papers.”
BackgroundMethodInfluentialSustainability Product Portfolio: A Review
Villamil, C., & Hallstedt, S. I. (2018) · European Journal of Sustainable Development · doi:10.14207/ejsd.2018.v7n4p146 · Cited by 20
“A snowballing method (Wohlin, 2014) was used to add relevant publications.”
MethodLezak, S., Ahearn, A., McConnell, F., & Sternberg, T. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.118099 · Cited by 28
“…following the familiarization and thematic framework development: 1) a semistructured “snowball” search of academic and grey literature documents (Wohlin, 2014); 2) a structured database and search-engine search (Mahood et al., 2014); and 3) handsearching the publication lists of relevant…”
MethodA Systematic Literature Review in Multi-Agent Systems: Patterns and Trends
Falco, M., & Robiolo, G. (2019) · Latin American Computer Conference (CLEI) · doi:10.1109/clei47609.2019.235098 · Cited by 31
“c) Snowballing: Following [18], we first constructed Table 1 to put together the initial point of analysis.”
MethodInfluentialSimilarity-based analyses on software applications: A systematic literature review
Auch, M., Weber, M., Mandl, P., & Wolff, C. G. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110669 · Cited by 27
Predictive models of diabetes complications: protocol for a scoping review
Ndjaboué, R., Farhat, I., Ferlatte, C.-A., Ngueta, G., Guay, D., Delorme, S., Ivers, N. M., Shah, B. R., Straus, S. E., Yu, C. H. Y., et al. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01391-w · Cited by 22
“In addition, we will perform snowballing of reference lists of selected papers at the full-text screening stage [28].”
MethodQuality attributes in patterns related to microservice architecture: a Systematic Literature Review
Valdivia, J. A., Limón, X., & Cortés-Verdín, K. (2019) · International Conference in Software Engineering Research and Innovation (CONISOFT) · doi:10.1109/conisoft.2019.00034 · Cited by 20
“In addition to the automated search with the previous criteria and search query, a manual research was incorporated using the backward snowballing method [18].”
MethodPinciroli, F., Barros‐Justo, J. L., & Forradellas, R. (2020) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2020.10.029 · Cited by 36
Mashkoor, A., Egyed, A. F., Wille, R., & Stock, S. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2457 · Cited by 42
“However, a large number of found studies during snowballing is not so wrong because, according to Wohlin, 14 the possibility of noise in snowballing is less than using a digital library approach, and, by deduction, snowballing is a better approach than a digital library search for extending…”
BackgroundMethodMair‐Bauernfeind, C., Zimek, M., Lettner, M., Hesser, F., Baumgartner, R. J., & Stern, T. (2020) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-020-01800-1 · Cited by 22
“Using a snowballing approach means to screen the reference list of a paper or citations to the paper to gather additional relevant literature (Wohlin 2014).”
How youth define, consume, and evaluate news: Reviewing two decades of research
Duvekot, S., Valgas, C. M., de Haan, Y., & de Jong, W. (2024) · New Media & Society · doi:10.1177/14614448241262809 · Cited by 42
“Finally, using the snowball method, relevant research articles mentioned in other studies were collected (Wohlin, 2014).”
Network Digital Twins: A Systematic Review
Verdecchia, R., Scommegna, L., Picano, B., Becattini, M., & Vicario, E. (2024) · IEEE Access · doi:10.1109/access.2024.3453034 · Cited by 27
“In order to complement the set of primary studies identified with the initial automated search, we conduct an exhaustive bidirectional snowballing process [22].”
BackgroundMethodInfluentialAccessibility of low-code approaches: A systematic literature review
Khalajzadeh, H., & Grundy, J. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107570 · Cited by 26
Hunger, T., Arnold, M. G., Engesser, S., & van den Boogaart, K. G. (2025) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2025.145834 · Cited by 27
Sana, T. Z., Abdulla, S., Nag, A., Das, A., Hassan, M. M., Fiza, Z. Z., Karim, A., & Kabir, S. R. R. (2025) · IEEE Access · doi:10.1109/access.2025.3605165 · Cited by 31
“To address our research inquiries, we have implemented the snowballing methodology as presented by Wohlin [208] to ascertain pertinent studies.”
A Systematic Literature Review of Infrastructure Studies in SIGCHI
Lyu, Y., Cai, J., & Carroll, J. M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757545 · Cited by 31
“We applied Wohlin’s snowballing methods [257] for the additional search.”
Utilising CI environment for efficient and effective testing of NFRs
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106199 · Cited by 25
“After applying the snowballing procedure defined by Wohlin et al. [24] , we found two extra articles, which were not included in the original search result.”
MethodA Snowballing Literature Study on Test Amplification
Danglot, B. (., Vera-Pérez, O. (., Yu, Z. (., Zaidman, A., Monperrus, M., & Baudry, B. (2019) · LillOA (Université de Lille (University Of Lille)) · Cited by 19
Assessing sense of community at farmers markets: A systematic review
Russomanno, J., & Jabson, J. M. (2021) · Journal of Agriculture Food Systems and Community Development · doi:10.5304/jafscd.2021.102.032 · Cited by 25
“…review, an additional 15 articles were identified using a snowball technique of article reference sections (Horsley, Dingwall, & Sampson, 2011; Wohlin, 2014) that involved reviewing the reference lists of included publications to see if these references produce yet unidentified publications…”
MethodA Systematic Review on Software Robustness Assessment
Laranjeiro, N., Agnelo, J., & Bernardino, J. (2021) · ACM Computing Surveys · doi:10.1145/3448977 · Cited by 23
“As a complement to the online search and with the goal of enriching the set of selected studies, we also used snowballing [207].”
MethodAlves, R. R., & de Campos, F. C. (2021) · Exacta · doi:10.5585/exactaep.2021.16057 · Cited by 30
Gutiérrez, L. F. R., Aguilar, J., Toro, M., & Suescún, E. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111010 · Cited by 25
Barriers to the Adoption of Autonomous Vehicles in Rural Communities
Prioleau, D., Dames, P., Alikhademi, K., & Gilbert, J. E. (2020) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas50296.2020.9462192 · Cited by 28
A Systematic Assessment on Android Third-Party Library Detection Tools
Zhan, X., Liu, T., Liu, Y., et al. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3115506 · Cited by 19
“To investigate existing TPL detection techniques, we first follow a well-defined Systematic Literature Review (SLR) methodology [58], [59] to find related research in this area.”
MethodZiogas, I. A., Kakos, C. D., Esagian, S. M., Skarentzos, K., Alexopoulos, S. P., Shingina, A., & Montenovo, M. I. (2021) · Clinical Transplantation · doi:10.1111/ctr.14521 · Cited by 23
Khan, F. M., Khan, J. A., Assam, M., Almasoud, A. S., Abdelmaboud, A., & Hamza, M. A. (2022) · IEEE Access · doi:10.1109/access.2022.3152073 · Cited by 30
“In SLR, the main goal is to answer what we have to do and how it will be achieved in the best way, along with the proofs and shreds of evidence [122].”
BackgroundKapsar, K. E., Frans, V. F., Brigham, L. W., & Liu, J. (2022) · AMBIO · doi:10.1007/s13280-022-01729-9 · Cited by 25
“To ensure that all relevant articles were identified, even if they were not in the search results, we conducted forward and backward reference checks using a snowballing method for all studies retained after initial screening (Wohlin 2014).”
MethodA Guideline for Contextual Adaptation of Community-Based Health Interventions
Pardoel, Z., Reijneveld, S. A., Postma, M. J., Lensink, R., Koot, J. A. R., Swe, K. H., Nguyen, M. V., Pamungkasari, E. P., Tenkink, L., Vervoort, J. P. M., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19105790 · Cited by 28
“To find the literature, we used the snowball method, using the bibliography or footnotes of a paper to identify additional papers [26].”
MethodAljedaani, W., Saad, E., Rustam, F., Díez, I. D. L. T., & Ashraf, I. (2022) · Mathematics · doi:10.3390/math10173199 · Cited by 23
Reporting of Freshwater Cyanobacterial Poisoning in Terrestrial Wildlife: A Systematic Map
Ash, A. K., & Patterson, S. (2022) · Animals · doi:10.3390/ani12182423 · Cited by 23
“An adapted version of Wohlin’s [43] snowballing method was performed after database results were screened against the inclusion criteria.”
MethodDolińska, W., Draper, H., Othman, L., Thompson, C., Girvan, S., Cunningham, K., Allen, J., Rigby, A. S., Phillips, K., & Guinn, B. (2022) · F&S Reviews · doi:10.1016/j.xfnr.2022.12.001 · Cited by 31
A Systematic Literature Review of Issue-Based Requirement Traceability
Lyu, Y., Cho, H., Jung, P., & Lee, S. (2023) · IEEE Access · doi:10.1109/access.2023.3242294 · Cited by 26
“We applied the forward and backward snowballing techniques to find additional papers [64].”
MethodMendoza, A., & Venables, A. T. (2023) · Journal of Information Technology Education Research · doi:10.28945/5082 · Cited by 48
“‘Snowballing’ or ‘chain - referral’ sampling is a non - probabilistic, “step - by - step” technique (Biernacki & Waldorf, 1981; Wohlin, 2014).”
A systematic literature review on automatic website generation
Kaluarachchi, T., & Wickramasinghe, M. (2023) · Journal of Computer Languages · doi:10.1016/j.cola.2023.101202 · Cited by 33
Analysing Interoperability in Digital Twin Software Architectures for Manufacturing
Ferko, E., Bucaioni, A., Pelliccione, P., & Behnam, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42592-9_12 · Cited by 33
“To reduce possible threats to construct validity we performed closed recursive snowballing activities [34].”
MethodAssessing sustainable production: A selection of indicators aligned with the SDGs
Viles, E., Cavallieri, M. S., Kalemkerian, F., Montoya‐Torres, J. R., & Santos, J. (2023) · Sustainable Development · doi:10.1002/sd.2793 · Cited by 25
Design Space of Visual Feedforward And Corrective Feedback in XR-Based Motion Guidance Systems
Yu, X., Lee, B., & Sedlmair, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642143 · Cited by 19
Inclusive Virtual Reality Learning: Review and 'Best-Fit' Framework for Universal Learning
Wehrmann, F., & Zender, R. (2024) · The Electronic Journal of e-Learning · doi:10.34190/ejel.21.6.3265 · Cited by 34
“To complement the initial search strategy and address the limited number of records, a backward snowballing technique (Wohlin, 2014) was used.”
Systems-of-systems development: Initiatives, trends, and challenges
Lana, C. A., de Souza, N. M., Delamaro, M. E., Nakagawa, E. Y., Oquendo, F., & Maldonado, J. C. (2016) · Latin American Computer Conference (CLEI) · doi:10.1109/clei.2016.7833329 · Cited by 26
“Furthermore, snowballing was used as described by Wholin [28].”
MethodBenefits and pitfalls applying RAMI4.0
Frysak, J., Kaar, C., & Stary, C. (2018) · doi:10.1109/icphys.2018.8387633 · Cited by 19
“Based on the references therein, related papers were searched using a snowballing based technique [14].”
MethodData mining in foreign language learning
Bravo, J., Bonilla, C. F., & Seoane, I. (2018) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1287 · Cited by 19
“Wohlin (2014) highlighted the usefulness of Google Scholar in systematic literature studies.”
BackgroundContextualizing Research Evidence through Knowledge Translation in Software Engineering
Badampudi, D., Wohlin, C., & Gorschek, T. (2019) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3319008.3319358 · Cited by 19
“Several guidelines have been proposed to conduct SLRs [28], [16] and [4] and to synthesize primary studies [10].”
BackgroundSelf-citationRequirement patterns: a tertiary study and a research agenda
Kudo, T. N., Bulcão-Neto, R. F., & Vincenzi, A. M. R. (2019) · IET Software · doi:10.1049/iet-sen.2019.0016 · Cited by 19
“The selection method is composed of inclusion criteria (IC) and EC [13], the snowballing technique [19], and quality assessment criteria (QC) [20].”
MethodConversational Interfaces for Health: Bibliometric Analysis of Grants, Publications, and Patents
Xing, Z., Yu, F., Du, J., Walker, J. S., Paulson, C. B., Mani, N. S., & Song, L. (2019) · Journal of Medical Internet Research · doi:10.2196/14672 · Cited by 25
“In addition, we used a snowball process to collect synonyms, term variations, and relevant terms to build a search term thesaurus for this study [27].”
Does the Mark-to-Model Fair Value Measure Make Assets Impairment Noisy?: A Literature Review
Dudycz, T., & Praźników, J. (2020) · Sustainability · doi:10.3390/su12041504 · Cited by 23
“Then, we decided to implement backward and forward snowballing to expand the data sample and review potential publications associated with the subject of research [22].”
Moroni, F., Spangaro, A., Carlino, M., Baber, U., Brilakis, E. S., & Azzalini, L. (2020) · International Journal of Cardiology · doi:10.1016/j.ijcard.2020.05.067 · Cited by 19
Systematic Literature Review on Test Case Selection and Prioritization: A Tertiary Study
Singhal, S., Jatana, N., Suri, B., Misra, S., & Sanz, L. F. (2021) · Applied Sciences · doi:10.3390/app112412121 · Cited by 24
“Using the references and the citations respectively are referred to as backward and forward snowballing” [16].”
BackgroundMethodInfluentialGunjur, A., Balasubramanian, A., Hafeez, U., Menon, S., Cher, L., Parakh, S., & Gan, H. K. (2022) · Journal of Neuro-Oncology · doi:10.1007/s11060-022-04092-7 · Cited by 19
A systematic mapping study about socio-technical congruence
Sierra, J. M., Vizcaíno, A., Genero, M., & Piattini, M. G. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.10.004 · Cited by 19
OntoGamif: A modular ontology for integrated gamification
Bouzidi, R., Nicola, A. D., Nader, F., & Chalal, R. (2019) · Applied Ontology · doi:10.3233/ao-190212 · Cited by 19
“At the end of the search, we carried out a backward and forward snowballing based on the guidelines defined by Wohlin (2014) for additional results (see Table 11).”
Microservice Architecture: A Tertiary Study
Costa, D. I. C., Filho, E. P. E. S., da Silva, R. F., Gama, T. D. D. C. Q., & Cortés, M. I. (2020) · Brazilian Symposium on Software Components, Architectures, and Reuse (SBCARS) · doi:10.1145/3425269.3425277 · Cited by 22
“Adi-cionalmente, foi aplicado o método snowballing [26] a partir dos trabalhos filtrados na primeira etapa, minimizando assim, as chances de qualquer estudo secundário relevante não ser identificado neste estudo.”
BackgroundA taxonomy of software bots: towards a deeper understanding of software bot characteristics
Lebeuf, C. (2018) · UVic’s Research and Learning Repository (University of Victoria) · Cited by 38
Blockchain Oracle Design PatternsPreprint
Pasdar, A., Dong, Z., & Lee, Y. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09349 · Cited by 55
“During the paper collection, snowballing technique [14] is employed to collect relevant studies for the literature review.”
MethodNavigating the DevOps landscape
Zhang, X., Zhao, P., & Jaskolka, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112331 · Cited by 21
Viollet, B., Scelles, N., & Peng, Q. (2023) · Sustainability · doi:10.3390/su15042949 · Cited by 18
Du, Y., Zhou, H., & Zha, W. (2024) · Journal of Health Population and Nutrition · doi:10.1186/s41043-024-00608-1 · Cited by 25
“To enhance the comprehensiveness of literature search, we conduced backward and forward snowballing searches to identify additional relevant articles [32].”
MethodSoftware solutions for newcomers’ onboarding in software projects: A systematic literature review
Santos, Í., Felizardo, K. R., Steinmacher, I., & Gerosa, M. A. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107568 · Cited by 41
“R1 conducted full snowballing, which identifies new studies based on the starting set, followed by backward and forward snow-balling, according to the guidelines for snowballing proposed in [121].”
MethodIntegration of blockchain and life cycle assessment: a systematic literature review
Zhang, L., & Fröhling, M. (2024) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-024-02371-1 · Cited by 22
“Finally, we completed the snowball process for the included papers (Wohlin 2014) and eventually found four related papers.”
Sleep disturbances in anorexia nervosa
Burger, P. C., Bos, R. W., Maas, J., Simeunovic-Ostojic, M., & Gemke, R. J. B. J. (2024) · European Eating Disorders Review · doi:10.1002/erv.3148 · Cited by 21
“A reference search was applied to capture any remaining eligible papers (Wohlin, 2014).”
A Survey on Flash-Memory Storage Systems: A Host-Side Perspective
Boukhobza, J., Olivier, P., Lim, W. S., Chen, L.-C., Hsieh, Y.-S., Wu, S.-T., Ho, C.-C., Huang, P., & Chang, Y.-H. (2025) · ACM Transactions on Storage · doi:10.1145/3723167 · Cited by 33
Wireless Communication in Underground Mining Teleoperation: A Systematic Review
Hasan, S., Sjöberg, K., & Wallin, P. (2025) · IEEE Access · doi:10.1109/access.2025.3563138 · Cited by 20
“To ensure a comprehensive and robust screening process, forward and backward snowballing [44] was conducted by examining the references and citations of the included studies, leading to the inclusion of 12 additional studies.”
Stability in Software Engineering: Survey of the State-of-the-Art and Research Directions
Salama, M., Bahsoon, R., & Lago, P. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2925616 · Cited by 27
Challenges and recommendations to publishing and using credible evidence in software engineering
Wohlin, C., & Rainer, A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106555 · Cited by 26
“The first author effectively conducted a forward snowballing [22] of citations.”
Self-citationThe orchestrator's partner management framework for software ecosystems
van Vulpen, P., Jansen, S., & Brinkkemper, S. (2021) · Science of Computer Programming · doi:10.1016/j.scico.2021.102722 · Cited by 22
Awareness Support in Collaborative System: Reviewing Last 10 Years of CSCW Research
Mantau, M. J., & Benitti, F. B. V. (2022) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd54268.2022.9776091 · Cited by 25
Kassim, S. A., Gartner, J., Labbé, L., Landa, P., Paquet, C., Bergeron, F., Lemaire, C., & Côté, A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-060357 · Cited by 23
Mugobo, V. V., Ntuli, H., & Iwu, C. G. (2022) · Journal of risk and financial management · doi:10.3390/jrfm15060244 · Cited by 23
“The list of studies was then extended using a snowball strategy of searching backward and forward citations (see Wohlin 2014) and again applying the above criteria.”
MethodAssessment and Test-case Study of Wi-Fi Security through the Wardriving Technique
Etta, V. O., Sarı, A., Imoize, A. L., Shukla, P. K., & Alhassan, M. (2022) · Mobile Information Systems · doi:10.1155/2022/7936236 · Cited by 22
“Snowballing is crucial for research since it leads to additional investigation [82].”
BackgroundMethodInfluentialRetractedGlass, T. W., Magoun, A. J., Robards, M. D., & Kielland, K. (2022) · Polar Biology · doi:10.1007/s00300-022-03079-4 · Cited by 22
“To accomplish this, we employed a snowballing procedure (Wohlin 2014), whereby we searched “wolverine* AND Arctic AND ‘Gulo gulo’” in Google Scholar, sorted in descending order of relevance and used the top 20 most relevant studies addressing wolverines north of latitudinal treeline as our start set.”
MethodKeeping the Parents outside the School Gate—A Critical Review
Kent, C., du Boulay, B., & Cukurova, M. (2022) · Education Sciences · doi:10.3390/educsci12100683 · Cited by 43
“These papers’ references were additionally checked for relevance using a forward snowballing approach [21].”
MethodAguayo, O., & Sepúlveda, S. (2022) · Applied Sciences · doi:10.3390/app122010240 · Cited by 23
How do Java mutation tools differ?
Amalfitano, D., Paiva, A. C. R., Inquel, A., Pinto, L. M. G., Fasolino, A. R., & Just, R. (2022) · Communications of the ACM · doi:10.1145/3526099 · Cited by 26
“First, we performed a snowballing procedure [43].”
Rashidian, S., Drogemuller, R., Omrani, S., & Banakar, F. (2023) · Smart and Sustainable Built Environment · doi:10.1108/sasbe-10-2022-0236 · Cited by 27
Introducing contextual transparency for automated decision systems
Sloane, M., Solano-Kamaiko, I. R., Yuan, J., Dasgupta, A., & Stoyanovich, J. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00623-7 · Cited by 28
“This approach uses a selection of known key papers on the topic and then deploys forward and backward snowballing to map the topical field and construct the literature review 65,66 .”
MethodBusiness process modeling language selection for research modelers
Farshidi, S., Kwantes, I. B., & Jansen, S. (2023) · Software & Systems Modeling · doi:10.1007/s10270-023-01110-8 · Cited by 34
“…nature of the studied area [77]; (2) Snowballing has been shown to be comparable to multiple database searches [80]; and (3) Snowballing is a promising approach for identifying relevant work, as new studies almost always cite at least one paper from previous but relevant studies [79].”
Artificial Intelligence in MOBA Games: A Multivocal Literature Mapping
Costa, L. M., Drachen, A., Souza, F. C. M., & Xexeo, G. B. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2023.3282157 · Cited by 26
“To ensure the inclusion of all the relevant publications, we conducted backward snowballing on the set of papers already in the pool, as recommended by systematic review guidelines [23].”
MethodFacial Emotion Recognition in Immersive Virtual Reality: A Systematic Literature Review
Ortmann, T., Wang, Q., & Putzar, L. (2023) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3594806.3594861 · Cited by 23
“Figure 1: Adapted PRISMA [20] procedure with additional snowballing [34]”
MethodOlari, V., Tenório, K., & Romeike, R. (2023) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-43393-1_21 · Cited by 27
“Snowballing refers to using the reference list of a paper or the citations to the paper to identify additional relevant papers [16].”
BackgroundArtificial Intelligence on Diagnostic Aid of Leprosy: A Systematic Literature Review
Fernandes, J. R. N., Teles, A. S., Fernandes, T. R. S., Lima, L. D. B., Balhara, S., Gupta, N., & Teixeira, S. S. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm13010180 · Cited by 26
“We finally performed the snowballing technique [78,79] to maximize results in the selection process.”
MethodA systematic literature review on the application of process mining to Industry 4.0
Akhramovich, K., Serral, E., & Cetina, C. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-023-02042-x · Cited by 32
“After the primary studies were selected, they were used as the starting set for backward and forward snowballing, which was performed according to the guidelines provided by Wohlin [44].”
Factors impacting the adoption of big data in healthcare: A systematic literature review
Teneiji, A. S. A., Salim, T. Y. A., & Riaz, Z. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105460 · Cited by 28
A theoretical framework to support green agripreneurship avoiding greenwashing
Mendes, J. A. J., Oliveira, A. Y., Santos, L., Gerólamo, M. C., & Zuin, V. G. (2024) · Environment Development and Sustainability · doi:10.1007/s10668-024-04965-z · Cited by 23
Martino, V. D., & Palomba, F. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107678 · Cited by 32
Searching for common ground: existing literature on automotive agile software product lines
Hohl, P., Ghofrani, J., Münch, J., Stupperich, M., & Schneider, K. A. (2017) · International Conference on Software and System Process · doi:10.1145/3084100.3084109 · Cited by 18
How Has the Health of Software Ecosystems Been Evaluated?
Amorim, S. D. S., Neto, F. S. D. S., McGregor, J. D., de Almeida, E. S., & Chávez, C. (2017) · Brazilian Symposium on Software Engineering · doi:10.1145/3131151.3131174 · Cited by 22
“• ACM Digital Library • IEEEXplore • ISI Web of Science • ScienceDirect • SpringerLink • Scopus In addition, we used the snowballing technique [22] and handsearched relevant papers for two events: international conference proceedings on software business (ICSOB) and international workshop on software ecosystems (IWSEC…”
MethodApplying Security Testing Techniques to Automotive Engineering
Pekaric, I., Sauerwein, C., & Felderer, M. (2019) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3339252.3340329 · Cited by 21
“We based our literature review by following two different methodologies: a keyword search [23] combined with forward and backward snowballing [25].”
BackgroundMethodHow do we design for concreteness fading?: survey, general framework, and design dimensions
Suh, S., Lee, M., & Law, E. (2020) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3392063.3394413 · Cited by 31
“, “snowballing” approach [89]) to end up with 673 papers.”
Background
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.