Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 1,601–1,700 of 3,934
Famiglietti, J., Acconito, L., Arpagaus, C., & Toppi, T. (2025) · Energy Conversion and Management X · doi:10.1016/j.ecmx.2025.100947 · Cited by 15
Software testing for extended reality applications: a systematic mapping study
Gu, R., Rojas, J. M., & Shin, D. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00523-7 · Cited by 24
SoK: Software Compartmentalization
Lefeuvre, H., Dautenhahn, N., Chisnall, D., & Olivier, P. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00075 · Cited by 41
“To cover works prior to 2003 and from other venues, we apply snowballing [250] to each paper of our set to identify 66 further works, and complete the list with 49 more from our knowledge of the industry and literature, totaling 211 works.”
The Digital Business Architect – Towards Method Support for Digital Innovation and Transformation
Wißotzki, M., & Sandkuhl, K. (2017) · Lecture notes in business information processing · doi:10.1007/978-3-319-70241-4_24 · Cited by 16
“In general, two methods are used: The „systematic method“ [6] and the snowballing method [7, 8], whereas for the initialization of the literature review, we opted the second approach.”
Mainey, L., Taylor, A., Baird, K. M., & O’Mullan, C. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0637-x · Cited by 13
“Each iteration of this entire process will be completed separately to maintain repeatability and traceability [26].”
BackgroundMethodInfluentialda Silva, J. P. S., Ecar, M., Pimenta, M. S., Guedes, G. T. A., Franz, L. P., & Marchezan, L. (2018) · International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) · doi:10.1145/3194133.3194136 · Cited by 14
“The snowballing technique refers to using study references and citations to identify additional studies [44].”
BackgroundMethodA tertiary study on model-based testing areas, tools and challenges: Preliminary results
Villalobos-Arias, L., López, C. U. Q., Porras, A. M., & Coronas, M. J. (2018) · Conferencia Iberoamericana de Software Engineering · Cited by 16
Social Aspects and How They Influence MSECO Developers
Steglich, C., Marczak, S. D. S., de Souza, C. R. B., Guerra, L. P., Mosmann, L. H., Filho, F. F., & Perin, M. G. (2019) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2019.00032 · Cited by 14
“This study had some limitations in its execution, however, the work was conducted with the maximum possible rigor, respecting the methodological processes used, as described by these authors [10], [11].”
BackgroundMethodApproach of Agile Methodologies in the Development of Web-Based Software
Ríos, J. R. M., & Pedreira, N. (2019) · Information · doi:10.3390/info10100314 · Cited by 21
“For the process of selecting primary studies and identifying relevant research, the snowball technique applied in Wohlin’s research [21] was used.”
MethodProfiling users via their reviews: an extended systematic mapping study
Dong, X. L., Li, T., Song, R., & Ding, Z. (2020) · Software & Systems Modeling · doi:10.1007/s10270-020-00790-w · Cited by 14
“Secondly, in order to expand the potentially related research, we use the “snowball” method to search for relevant literature manually [37].”
Method30 Years of Software Refactoring Research:A Systematic Literature Review
Abid, C., Alizadeh, V., Kessentini, M., Ferreira, T., & Dig, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02194 · Cited by 79
Nkrumah, R. B., & Sinha, V. (2020) · Review of Education · doi:10.1002/rev3.3205 · Cited by 17
Impact of programming languages on energy consumption for mobile devices
Kholmatova, Z. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3418777 · Cited by 13
Determinants of adoption in open-source hardware: A review of small wind turbines
Reinauer, T., & Hansen, U. E. (2021) · Technovation · doi:10.1016/j.technovation.2021.102289 · Cited by 17
Requirements prioritization based on multiple criteria using Artificial Intelligence techniques
Limaylla-Lunarejo, M.-I. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00072 · Cited by 16
Software Visualizations to Analyze Memory Consumption: A Literature Review
Blanco, A. F., Bergel, A., & Alcocer, J. P. S. (2022) · ACM Computing Surveys · doi:10.1145/3485134 · Cited by 18
“The snowballing procedure consists of identifying additional studies using the system of references between articles [105].”
MethodSystematic Literature Review of AI/ML Techniques Applied to VANET Routing
Teixeira, D., Ferreira, J., & Macedo, J. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-030-98015-3_23 · Cited by 11
Matias, G. D. S., Lermen, F. H., Bissaro, C. A., Nicolin, D. J., Fischer, C., & Jorge, L. M. D. M. (2022) · Journal of Food Process Engineering · doi:10.1111/jfpe.14060 · Cited by 15
Personalised immunotherapy in sepsis: a scoping review protocol
Slim, M. A., van Mourik, N., Dionne, J. C., Oczkowski, S. J. W., Netea, M. G., Pickkers, P., Giamarellos‐Bourboulis, E. J., Müller, M. C. A., van der Poll, T., Wiersinga, W. J., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-060411 · Cited by 16
Katey, D., Addo, A. A., Abass, K., & Morgan, A. K. (2022) · Journal of Endocrinology Metabolism and Diabetes of South Africa · doi:10.1080/16089677.2022.2074121 · Cited by 17
Belle, A. B., & Zhao, Y. (2022) · Applied Sciences · doi:10.3390/app12189017 · Cited by 14
“That analysis also focused on the reporting procedures discussed by Kitchenham [32], Petticrew and Roberts [33], Kitchenham [34], Pickering and Byrne [35], Wohlin [36], and Okoli [30].”
BackgroundMethodEmpirical Formal Methods: Guidelines for Performing Empirical Studies on Formal Methods
ter Beek, M. H., & Ferrari, A. (2022) · Software · doi:10.3390/software1040017 · Cited by 16
Alves, J., Lima, T. M., & Gaspar, P. D. (2023) · Designs · doi:10.3390/designs7010023 · Cited by 12
Robotic Process Automation and Intelligent Automation Security Challenges: A Review
Al-Slais, Y., & Ali, M. (2023) · International Conference on Cyber Management and Engineering (CyMaEn) · doi:10.1109/cymaen57228.2023.10050996 · Cited by 23
“Snowballing, according to Wohlin [46], refers to the utilization of a publication’s reference list or citations to identify additional publications for systematic literature reviews.”
Jalo, H., Seth, M., Pikkarainen, M. A., Häggström, I., Jood, K., Bakidou, A., Sjöqvist, B. A., & Candefjord, S. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-069660 · Cited by 18
Best Practice Mechanisms for Biodiversity Conservation Law and Policy
Brockett, C., Woolaston, K., Deane, F., Humphries, F., Kumar, E., Kennedy, A., & Bell‐James, J. (2023) · Cambridge Prisms Extinction · doi:10.1017/ext.2023.14 · Cited by 11
Mossige, M., Arendal, E., Kongskov, L., & Svendsen, H. B. (2023) · Dyslexia · doi:10.1002/dys.1752 · Cited by 16
An Analysis of Methods and Metrics for Task Scheduling in Fog Computing
Misirli, J., & Casalicchio, E. (2023) · Future Internet · doi:10.3390/fi16010016 · Cited by 15
Vertical Test Reuse for Embedded Systems: A Systematic Mapping Study
Flemström, D., Sundmark, D., & Afzal, W. (2015) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2015.46 · Cited by 18
“In order to make sure we do not miss any relevant papers prior to year 2005, we added a forward-backward snowballing stage [24] for the 15 references.”
BackgroundModel-Driven Development in Robotics Domain: A Systematic Literature Review
Heineck, T., Gonçalves, E. J. T., de Sousa, A. M. C., Oliveira, M., & Castro, J. (2016) · Brazilian Symposium on Software Components, Architectures, and Reuse (SBCARS) · doi:10.1109/sbcars.2016.12 · Cited by 13
Program comprehension through reverse‐engineered sequence diagrams: A systematic review
Ghaleb, T. A., Alturki, M. A., & Aljasser, K. (2018) · Journal of Software Evolution and Process · doi:10.1002/smr.1965 · Cited by 12
“62 Moreover, Wohlin 63 also encourages to perform a snowballing search 64 as a complementary search to reduce the likelihood of missing important literature, which makes the literature review more reliable.”
BackgroundModel Management Tools for Models of Different Domains: A Systematic Literature Review
Torres, W., van den Brand, M., & Serebrenik, A. (2019) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon.2019.8836869 · Cited by 14
“, snowballing [18], [19], systematic literature review (SLR) [20], and systematic mapping review (SMR) [21].”
BackgroundMethodDefining agile requirements change management: a mapping study
Albuquerque, D. W., Guimarães, E. T., Perkusich, M., Costa, A., Dantas, E., Ramos, F., & de Almeida, H. O. (2020) · ACM Symposium on Applied Computing · doi:10.1145/3341105.3374095 · Cited by 19
“In the third step (Snowballing), which was also peer reviewer, we applied backward and forward snowballing iteratively following the guidelines defined in [30].”
MethodThe use of lean methodology in healthcare settings in developing countries: a narrative review
Kelendar, H., Faisal, M., McIntosh, B., & Mohammed, M. A. (2020) · British Journal of Healthcare Management · doi:10.12968/bjhc.2019.0095 · Cited by 17
Ziogas, I. A., Johnson, W. R., Matsuoka, L. K., Rauf, M. A., Thurm, C. W., Hall, M., Bacchetta, M. D., Godown, J. A., & Alexopoulos, S. P. (2020) · Transplantation · doi:10.1097/tp.0000000000003414 · Cited by 12
Ahmad, B., Richardson, I., & Beecham, S. (2020) · doi:10.1109/rewbah51211.2020.00010 · Cited by 17
“As the first step in the research process, a literature review was conducted following the guidelines presented by Wohlin, 2014 [44].”
MethodQuality evaluation models or frameworks for open source software: A systematic literature review
Yılmaz, N., & Tarhan, A. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2458 · Cited by 22
“Finally, in order to improve the coverage, backward snowballing and forward snowballing 27 were applied, and four more studies were added to the 3rd set of publications.”
MethodA Systematic Literature Review of Crowdsourcing-Based Research in Information Security
Sauerwein, C., Gander, M., Felderer, M., & Breu, R. (2016) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose.2016.67 · Cited by 15
“According to [41] there is no general methodology for defining it.”
BackgroundMethodInfluentialBecker, C. A., Lorig, F., & Timm, I. J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-12738-1_2 · Cited by 12
LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead
He, J., Shi, J., Zhuo, T. Y., Treude, C., Sun, J., Du, X., Xing, Z., & Lo, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3797276 · Cited by 12
“For our literature search, we selected the Digital Bibliography & Library Project (DBLP) computer science bibliography 1 as the primary data source.”
A Scoping Review of Informed Consent Practices in Human–Computer Interaction Research
Schwind, V., Tadesse, N. Z., da Cunha, E. S., Hamidi, Y., Sultani, S. S., & Sehrt, J. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3721284 · Cited by 18
Carbon-Efficient Software Design and Development: A Systematic Literature Review
Danushi, O., Forti, S., & Soldani, J. (2025) · ACM Computing Surveys · doi:10.1145/3728638 · Cited by 39
“The list of candidate studies was completed by running backward snowballing [115], which resulted in 30 additional candidate studies.”
da Costa, M. G., de Abreu, V. H. S., de Assis, T. F., da Costa, V. X., D’Agosto, M. D. A., & Souza, A. D. S. (2022) · Environmental footprints and eco-design of products and processes · doi:10.1007/978-981-19-7226-3_5 · Cited by 17
The effect of ageing on confrontation naming in healthy older adults: a three-level meta-analysis
Wen, H., & Dong, Y. (2023) · Journal of Cognitive Psychology · doi:10.1080/20445911.2023.2184745 · Cited by 12
“Moreover, we applied a snowballing procedure to identify studies (see Wohlin, 2014), i.e. a backward snowballing strategy to identify possible studies from the reference lists of the included 484 H. WEN AND Y. DONG studies, and a forward snowballing strategy to search studies that cited the…”
MethodAbrar, M. F., Khan, M. S., Khan, I. U., Ali, G., & Shah, S. (2023) · Applied Sciences · doi:10.3390/app13074483 · Cited by 25
Southwest US winter precipitation variability: reviewing the role of oceanic teleconnections
Karanja, J., Svoma, B. M., Walter, J., & Georgescu, M. (2023) · Environmental Research Letters · doi:10.1088/1748-9326/accd84 · Cited by 11
People's migrations and plants for food: a review for fostering sustainability
Hanazaki, N., Pieroni, A., Ludwinsky, R. H., Gonçalves, M. C., Prakofjewa, J., Peroni, N., & Söukand, R. (2023) · Sustainable Earth Reviews · doi:10.1186/s42055-023-00058-3 · Cited by 15
“For this set of included studies, we did a snowball literature sampling [16], checking for the cited articles in each study adherent to our inclusion criteria.”
Theben, A., Djourova, N. P., & Freire, A. (2023) · Management Review Quarterly · doi:10.1007/s11301-023-00387-9 · Cited by 25
“In addition to the database search, a manual search was performed using a back-ward and forward snowballing procedure by searching the bibliographic lists of the selected studies for additional works of relevance (Wohlin 2014; Neumann 2021; Theres & Strohmeier 2023).”
Visual political communication research: A literature review from 2012 to 2022
Farkas, X. (2023) · Journal of Visual Political Communication · doi:10.1386/jvpc_00027_1 · Cited by 24
Saywell, I., Foreman, L. M., Child, B., Phillips‐Hughes, A. L., Collins‐Praino, L. E., & Baetu, I. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105672 · Cited by 14
“An additional 15 records were added through forward and backward snowballing methods (Wohlin, 2014).”
MethodA Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track
Alshahrani, M. H., & Maashi, M. S. (2024) · IEEE Access · doi:10.1109/access.2024.3404056 · Cited by 18
“To broaden the range of our initial search results, we carried out a round of backward snowballing, following the method described by Wohlin [15].”
MethodCharacterizing UX Assessment in the Context of Immersive Experiences: A Systematic Mapping Study
Marques, L., Barcellos, M. P., Gadelha, B., & Conte, T. U. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2351711 · Cited by 20
“The results of search strings are heavily dependent on their quality (Wohlin, 2014; Zhang et al., 2011).”
Perera, J., Tempero, E. D., Tu, Y., & Blincoe, K. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3675393 · Cited by 29
“Backward reference snowballing [64] was conducted on the included articles as an additional step to avoid the possibility of missing articles through only conducting a database search [50].”
Inclusion in Virtual Reality Technology: A Scoping Review
Yong, X., Chan, G., & Arya, A. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2392967 · Cited by 23
“To capture the full picture of the field of VR with respect to inclusion and our research questions, we conducted several rounds of literature search by employing a snowball method (Wohlin, 2014).”
Tertiary Review on Explainable Artificial Intelligence: Where Do We Stand?
van Mourik, F., Jutte, A., Berendse, S. E., Bukhsh, F. A., & Ahmed, F. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6030098 · Cited by 20
Esagian, S. M., Karam, J. A., Msaouel, P., & Makrakis, D. (2024) · European Urology Focus · doi:10.1016/j.euf.2024.08.002 · Cited by 14
Route choice modelling for an urban rail transit network: past, recent progress and future prospects
Tian, Y., Zhu, W., & Song, F. (2024) · European Transport Research Review · doi:10.1186/s12544-024-00677-7 · Cited by 16
Cosoveanu, F. S., Hegger, D. L. T., Mees, H. L. P., Buijs, J., Terpstra, T., & Driessen, P. (2024) · Environmental Policy and Governance · doi:10.1002/eet.2127 · Cited by 13
Digital marketplaces in European research landscape: A systematic literature review
Nikoletos, G., Papoutsoglou, I., Spanos, G., Nizamis, A., Lalas, A. X., Votis, K., & Tzovaras, D. K. (2024) · Open Research Europe · doi:10.12688/openreseurope.18657.1 · Cited by 14
Cybersecurity Testing in Drones Domain: A Systematic Literature Review
Marchetti, E., Waheed, T., & Calabrò, A. (2024) · IEEE Access · doi:10.1109/access.2024.3495994 · Cited by 12
“Quality Analysis 3rd Step: (See Figure 1) The accepted papers were manually enriched through a snowballing search [50].”
Gkargkavouzi, A., & Halkos, G. (2024) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2024.102510 · Cited by 19
Learning-based estimation of cattle weight gain and its influencing factors
Hossain, M. R. H., Islam, M. R., McGrath, S. R., Islam, M. Z., & Lamb, D. W. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110033 · Cited by 18
Compliance risks in the construction of residential buildings: a systematic literature review
Ghafoor, S., Gurmu, A. T., Sadick, A.-M., et al. (2025) · Smart and Sustainable Built Environment · doi:10.1108/sasbe-11-2024-0507 · Cited by 11
Hosseini, S. T. M., Qayumi, K. A., Pourabbasi, A., Haghighi, E., Sabet, B., Koohpaei, A., Shafiei, Z., Hosseini, M. M., & Nemati, P. (2025) · Discover Education · doi:10.1007/s44217-025-00521-7 · Cited by 26
Static Code Analysis for IoT Security: A Systematic Literature Review
Gomes, D. R., Felix, E. F., Aires, F., & Vieira, M. P. A. (2025) · ACM Computing Surveys · doi:10.1145/3745019 · Cited by 24
“The snowballing process relies on reviewing relevant studies to identify additional ones [58, 120, 121].”
Wen, Y., Wang, Z., & Guo, X. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2524029 · Cited by 18
“Furthermore, we employed snowballing to identify additional papers that might have been missed by the automated search (Wohlin, 2014).”
Software-Intensive Product Engineering in Start-Ups: A Taxonomy
Klotins, E., Unterkalmsteiner, M., & Gorschek, T. (2018) · IEEE Software · doi:10.1109/ms.2018.2801548 · Cited by 13
A Systematic Mapping of Software Engineering Approaches to Develop Big Data Systems
Laigner, R., Kalinowski, M., Lifschitz, S., Monteiro, R. S., & de Oliveira, D. (2018) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2018.00079 · Cited by 18
“Nevertheless, besides conducting the search on several digital libraries our search strategy also involved applying backward and forward snowballing on the initially included papers (seed set) to identify additional relevant papers that, for any reason, did not match our search string [74][53].”
BackgroundMethodAffia, A.-A. O., Matulevičius, R., & Nolte, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33246-4_18 · Cited by 13
“Also, we conducted manual searches [3,50] to complement the search procedure.”
MethodE-assessment and Academic Integrity: A Literature Review
Tsigaros, T., & Fesakis, G. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-73988-1_24 · Cited by 11
Understanding the context of IoT software systems in DevOps
Pereira, I. M., Carneiro, T. G. D. S., & Figueiredo, E. (2021) · IEEE/ACM International Workshop on Software Engineering Research and Practices for the IoT (SERP4IoT) · doi:10.1109/serp4iot52556.2021.00009 · Cited by 16
“In our SLR, we run the backward snowballing [17].”
MethodJandrey, A., Ruiz, D., & Silveira, M. (2021) · Simpósio Brasileiro de Fatores Humanos em Sistemas Computacionais · doi:10.1145/3472301.3484356 · Cited by 13
“Para o presente trabalho, utilizou-se a técnica de Snowballing seguindo os procedimentos propostos por Wohlin [37].”
BackgroundInfluentialChristoforou, A. P., Andreou, A. S., Garriga, M., & Baresi, L. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.108066 · Cited by 15
Attraction Basins in Metaheuristics: A Systematic Mapping Study
Baketarić, M., Mernik, M., & Kosar, T. (2021) · Mathematics · doi:10.3390/math9233036 · Cited by 12
Exploratory Study on Modelling Agricultural Carbon Emissions in Ireland
Madden, S. M., Ryan, A., & Walsh, P. (2021) · Agriculture · doi:10.3390/agriculture12010034 · Cited by 12
Singh, A., Hauge, J. B., & Wiktorsson, M. (2021) · Sustainability · doi:10.3390/su14010017 · Cited by 12
“In backward snowballing, we scan the articles cited in the initially selected articles whereas in forward snowballing we identify the articles citing the initially selected articles [33,34].”
MethodAdams, S. T., Bedwani, N. H., Massey, L. H., Bhargava, A., Byrne, C., Jensen, K. K., Smart, N. J., & Walsh, C. J. (2022) · Hernia · doi:10.1007/s10029-022-02562-5 · Cited by 16
“In response to this low yield it was agreed by the authors that the scope of the study should be widened to additionally include any papers identified via a second search performed by LM and NHB using the snowballing technique as described by Wohlin [14].”
MethodGarlasco, J., Nurchis, M. C., Bordino, V., Sapienza, M., Altamura, G., Damiani, G., & Gianino, M. M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19084862 · Cited by 15
Gender equality in tech entrepreneurship: a systematic mapping study
Wilson, A. W., & Patón-Romero, J. D. (2022) · doi:10.1145/3524501.3527608 · Cited by 17
“Likewise, it is important to highlight that an additional search was conducted in addition to the search done in Scopus , using the backward snowballing technique [25].”
Characterizing UX Evaluation in Software Modeling Tools: A Literature Review
Kalantari, R., & Lethbridge, T. C. (2022) · IEEE Access · doi:10.1109/access.2022.3227504 · Cited by 11
“We used Google Scholar to identify the start set, which Wohlin’s [35] recommends as a good choice to avoid pub-lisher bias.”
BackgroundMethodInfluentialLearning analytics in programming courses: Review and implications
Omer, U., Tehseen, R., Farooq, M. S., & Abid, A. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11611-0 · Cited by 19
“1 Searching through snowballing In addition to searching the relevant studies on different research portals by using the devised search string, another search cycle has been carried out through backward snowballing (Wohlin, 2014).”
BackgroundExtracorporeal life support in thoracic emergencies—a narrative review of current evidence
Willers, A., Mariani, S., Maessen, J. G., Lorusso, R., & Swol, J. (2023) · Journal of Thoracic Disease · doi:10.21037/jtd-22-1307 · Cited by 11
“Reference lists of assessed full texts were screened for further relevant studies (16).”
Virtual Tours to Facilities for Educational Purposes: A Review
Cardona, H., Lara-Álvarez, C. A., Parra, E., & Villalba‐Condori, K. O. (2023) · TEM Journal · doi:10.18421/tem123-55 · Cited by 25
“in 2014 [9], involves systematically exploring references and citations within collected literature.”
BackgroundAn Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code
Hort, M., Grishina, A., & Moonen, L. (2023) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem56168.2023.10304803 · Cited by 18
“We make small adjustments to the search procedure described by Wohlin et al. [33], as we aim to build on four recent surveys in the domain of deep learning models for software engineering tasks.”
MethodVidal-Durango, J. V., Baena-Navarro, R., & Therán-Nieto, K. (2024) · Indonesian Journal of Electrical Engineering and Computer Science · doi:10.11591/ijeecs.v34.i2.pp726-744 · Cited by 11
Wiapo, C., Adams, S., Komene, E., Davis, J., & Clark, T. C. (2024) · Journal of Clinical Nursing · doi:10.1111/jocn.17205 · Cited by 16
Visualization for Software Product Lines: A Systematic Mapping Study
Lopez-Herrejon, R. E., Illescas, S., & Egyed, A. F. (2016) · IEEE Working Conference on Software Visualization (VISSOFT) · doi:10.1109/vissoft.2016.11 · Cited by 13
“As part of our future work, we want to take a closer look on the interaction capabilities on the identified approaches and analyze the empirical foundations on which they rely upon.”
BackgroundSoftware Engineering Aspects of Green and Sustainable Software: A Systematic Mapping Study
Marimuthu, C., & Chandrasekaran, K. (2017) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3021460.3021464 · Cited by 26
“We have conducted this study to answer the seven different research questions and snowballing [22] method was used to collect the relevant studies from the literature.”
BackgroundMethodInfluential“I’m Blind, Can I Play?” Recommendations for the Development of Audiogames
Borges, O. T., & Campos, M. D. B. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-58703-5_26 · Cited by 11
“Thus, a literature review based on the Snowballing technique [28] was performed to identify guidelines specific to the development of accessible games.”
MethodSoftware engineering education in Brazil
da Cunha, J. A. O. G., Marques, G. A., Lemos, W. R., et al. (2018) · doi:10.1145/3266237.3266259 · Cited by 11
“Para complementar a busca automática e manual, foram realizadas rodadas de snowballing [18].”
MethodCatalog of Invisibility Requirements for UbiComp and IoT Applications
Carvalho, R. M., Andrade, R. M. C., de Oliveira, K. M., et al. (2018) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2018.00019 · Cited by 11
“Snowballing refers to using the reference list of a paper (backward snowballing) or citations to that paper (forward snowballing) to identify additional papers [35].”
BackgroundMethodSystematic Literature Review on User Logging in Virtual Reality
Luoto, A. (2018) · International Academic Mindtrek Conference (Mindtrek) · doi:10.1145/3275116.3275123 · Cited by 13
“One way to improve the situation could be to encourage the VR researchers who log users to discuss the used logging procedure more in detail.”
BackgroundFrank, A. S. K., & Schäffler, L. (2019) · Sustainability · doi:10.3390/su11205695 · Cited by 16
“We conducted a modified “snowball principle” based literature search using any paper which yielded relevant knowledge gaps to identify further relevant papers, as well as to check for subsequent citing publications [79,80].”
MethodHassanzadeh, H., Nguyen, A. N., & Verspoor, K. M. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103321 · Cited by 13
Load Balancing in the Presence of Services in Named-Data Networking
Mansour, D., Osman, H., & Tschudin, C. (2019) · Journal of Network and Systems Management · doi:10.1007/s10922-019-09507-x · Cited by 12
“For each of these interesting domains, we identify few main studies using searches on Google Scholar, then perform Snowballing [10] to gather further studies.”
Specifying the Process Model for Systematic Reviews: An Augmented Proposal
Olsina, L. A., Becker, P., Peppino, D., & Tebes, G. (2019) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2019.460 · Cited by 16
Search-based fault localisation: A systematic mapping study
Leitao-Junior, P. S., de Freitas, D. M., Vergílio, S. R., Camilo-Junior, C. G., & Harrison, R. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106295 · Cited by 11
“The FS used the Google Scholar search engine due to its great capability of searching citations across many publishers available [35].”
MethodThe role of canids in the dispersal of commercial and ornamental palm species
Spennemann, D. H. R. (2020) · Mammal Research · doi:10.1007/s13364-020-00535-6 · Cited by 15
Ahmad, A., Barros‐Justo, J. L., Feng, C., & Khan, A. A. (2020) · Applied Sciences · doi:10.3390/app10217749 · Cited by 14
“In addition to the automated search strategy, backward and forward snowballing [40,41] were performed as a complementary search strategies.”
MethodBreakthrough and Disruptive Innovation: A Theoretical Reflection
Rakic, K. (2020) · Journal of technology management & innovation · doi:10.4067/s0718-27242020000400093 · Cited by 21
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.