Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 2,101–2,200 of 3,934
Mylonas, K. S., Ziogas, I. A., Mylona, C. S., et al. (2020) · World Journal of Cardiology · doi:10.4330/wjc.v12.i11.540 · Cited by 6
Have Variability Tools Fulfilled the Needs of the Software Industry?
Allian, A. P., OliveiraJr, E., Capilla, R., et al. (2020) · JUCS - Journal of Universal Computer Science · doi:10.3897/jucs.2020.067 · Cited by 6
“Furthermore, each of the 79 studies was carefully reviewed by applying the snowballing technique [Wohlin, 2014, Wohlin et al., 2012].”
MethodAppropriate Assembly Instruction Modes: Factors to Consider
Kuipers, N., Kolbeinsson, A., & Thorvald, P. (2021) · Advances in transdisciplinary engineering · doi:10.3233/atde210007 · Cited by 6
Smart City Replication and Group Model Building: A Conceptual Comparison
Ruess, P. (2021) · IEEE European Technology and Engineering Management Summit (E-TEMS) · doi:10.1109/e-tems51171.2021.9524862 · Cited by 6
Blockchain’s potential and opportunities for tax administrations: a systematic review
Nascimento, L. D. V., Silva, P. D., & Peres, C. B. (2021) · International Conference on Blockchain Computing and Applications (BCCA) · doi:10.1109/bcca53669.2021.9657036 · Cited by 10
Investigando Colaboração em Ecossistemas
Pinheiro, M., Chueri, L., & dos Santos, R. P. (2021) · doi:10.5753/washes.2021.15885 · Cited by 8
Micro‐IDE: A tool platform for generating efficient deployment alternatives based on microservices
Aksakalli, I. K., Çelik, T., Can, A. B., et al. (2022) · Software Practice and Experience · doi:10.1002/spe.3088 · Cited by 6
“36 To evaluate our tool, all of the microservice types used in real scenarios for the Spotify application case study could not be reached.”
MethodThe state of the art in measurement-based experiments on the mobile web
de Munk, O., Scoccia, G. L., & Malavolta, I. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106944 · Cited by 9
An Empirical Study on the Fairness of Pre-trained Word Embeddings
Sesari, E., Hort, M., & Sarro, F. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.15 · Cited by 11
“After selecting a set of relevant papers from the repository search, one level of backwards snow-balling (Wohlin, 2014) was done to examine their references.”
BackgroundA systematic mapping study of security concepts for configurable data storages
May, R., Biermann, C., Krüger, J., Saake, G., & Leich, T. (2022) · International Software Product Line Conference (SPLC) · doi:10.1145/3546932.3546994 · Cited by 9
“, analyzing the references of the already selected papers) [127].”
BackgroundAgbedanu, P. R., Musabe, R., Rwigema, J., Gatare, I., Maginga, T. J., & Amenyedzi, D. K. (2022) · F1000Research · doi:10.12688/f1000research.127732.1 · Cited by 7
Welzmueller, J., & Schmidt, S. L. (2022) · Management Review Quarterly · doi:10.1007/s11301-022-00307-3 · Cited by 8
“Applying the backward snowballing technique (Wohlin 2014) seemed to be a useful extension, leading to a semi-systematic review.”
A systematic mapping study of software performance research
Han, X., Yu, T., & Yan, G. (2023) · Software Practice and Experience · doi:10.1002/spe.3185 · Cited by 12
“Authors examine the related work and references (backward snowballing 20 ) to add any primary studies that may have been overlooked in the initial search.”
BackgroundHow Agile Organizations Use Metrics: A Systematic Literature Mapping
Leal, S., Hauck, J. C. R., Bertan, M., et al. (2022) · doi:10.1145/3571473.3571479 · Cited by 6
“As proposed in the guidelines for SLMs [44],[50],[64], we undertook a series of activities grouped into four main phases: (i) protocol definition, (ii) execution of searches, (iii) study selection and snowballing (iv) data extraction and analysis.”
MethodInfluentialA Maturity Model for Digital Business Ecosystems from an IT Perspective
Ehrensperger, R., Sauerwein, C., & Breu, R. (2023) · JUCS - Journal of Universal Computer Science · doi:10.3897/jucs.79494 · Cited by 13
Namoun, A., Tufail, A., Nawaz, W., Benrhouma, O., & Alshanqiti, A. M. (2023) · Computational Intelligence and Neuroscience · doi:10.1155/2023/5934548 · Cited by 6
“Although such manual approaches may be thought less efcient, the literature backs up their usefulness in pinpointing relevant articles [61].”
BackgroundMethodInfluentialCampos, L. M. D. S., Karl, A. A., & Vázquez‐Brust, D. (2023) · Greening of industry networks studies · doi:10.1007/978-3-031-25436-9_3 · Cited by 10
A Taxonomy of Assets for the Development of Software-Intensive Products and Services
Zabardast, E., González‐Huerta, J., Gorschek, T., Šmite, D., Alégroth, E., & Fagerholm, F. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111701 · Cited by 14
“For instance, from a managerial perspective where the term asset is used to discuss how products in product lines are developed from a set of core assets with built-in variation mechanisms [62], or making emphasis only on business- or market-related assets, like the in the works by Ampatzoglou et al. [4], Cicchetti et…”
MethodInfluentialA survey on automation approaches of smart contract generation
Mars, R., Cheikhrouhou, S., Kallel, S., & Kacem, A. H. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05262-8 · Cited by 10
“Finally, to extend the systematic search protocol defined earlier, we used the so-called snowballing technique [25].”
MethodA Review on Quality in Use Evaluation of Smart Environment Applications: What’s Next?
Angeloni, M. P. C., de Oliveira, K. M., Strugeon, E. G.-L., Duque, R., & Tîrnȃucȃ, C. (2023) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/3596454.3597177 · Cited by 7
“However, several studies have shown that combining Sco-pus and Web of Science with snowballing procedures can mitigate the lack of search engines and provide a representative set of studies to a characterization work [23, 38].”
BackgroundModelling guidance in software engineering: a systematic literature review
Chakraborty, S., & Liebel, G. (2023) · Software & Systems Modeling · doi:10.1007/s10270-023-01117-1 · Cited by 14
“In addition to this database search, we conducted a second search using more general terminology and random sampling among the 71450 resulting papers, and snowballing [99] in the area of Business Process Modelling to complement our findings.”
An IoT system for managing machine tool spindles in operation
Bachim, T., Martens, M. L., Gonçalves, R. F., Bizarrias, F. S., & Machado, M. C. (2023) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-023-11936-7 · Cited by 9
“According to Wohlin [79], utilizing snowballing as a search strategy can indeed be a viable substitute for database searches.”
MethodDalko, K., Kraft, B., Jahn, P., Schildmann, J., & Hofstetter, S. (2023) · Journal of Medical Internet Research · doi:10.2196/46297 · Cited by 10
“A literature review following a snowball approach based on the guidelines of Wohlin [24] was conducted to identify publications.”
Product-Structuring Concepts for Automotive Platforms
Zellmer, P., Holsten, L., Leich, T., et al. (2023) · ACM International Systems and Software Product Line Conference (SPLC) · doi:10.1145/3579027.3608988 · Cited by 6
“We extended and thereby improved the completeness of our search by performing backwards snowballing [46, 80].”
On transforming model‐based tests into code: A systematic literature review
Ferrari, F. C., Durelli, V. H. S., Andler, S. F., Offutt, J., Saadatmand, M., & Müllner, N. (2023) · Software Testing Verification and Reliability · doi:10.1002/stvr.1860 · Cited by 6
“We follow the guidelines for conducting secondary studies proposed by Kitchenham et al. [9] and Wohlin [10].”
BackgroundMethodInfluentialA Systematic Literature Review on Software Vulnerability Prediction Models
Bassi, D., & Singh, H. (2023) · IEEE Access · doi:10.1109/access.2023.3312613 · Cited by 6
“There lies a possibility that the search strategy might skip some of the relevant studies therefore manual search such as forward and backward snowballing is performed [14].”
Momin, M., & Ali, O. (2023) · HighTech and Innovation Journal · doi:10.28991/hij-2023-04-03-014 · Cited by 11
Huber, S., Lorey, T., & Felderer, M. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00051 · Cited by 6
“Our research process closely follows existing scientific guidelines on systematic literature reviews [11], [12] and principles on developing taxonomies [13].”
MethodGodsk, M., & Nielsen, B. L. (2024) · Nordic Journal of Systematic Reviews in Education · doi:10.23865/njsre.v2.5498 · Cited by 19
“The literature review was structured as a systematic literature review (Khan et al., 2003; Littell et al., 2008), guided by a PRISMA process (Moher et al., 2009; see Figure 1), supplemented with hand searches and a backward snowballing process (Wohlin, 2014).”
MethodLoufopoulos, G., Manaki, V., Tasoudis, P., et al. (2024) · Angiology · doi:10.1177/00033197241241788 · Cited by 6
Leveraging Value Creation Toward Business Models for Sustainability
Cabrita, M. D. R. (2023) · IEEE International Conference on Technology and Entrepreneurship (ICTE) · doi:10.1109/icte58739.2023.10488582 · Cited by 7
“Following the guidelines proposed by Wohlin [14,15] the literature selection process used to find the articles is shown in Table 1 The term “business model” made its initial appearance in an academic article in 1957 [16] with the concept gaining popularity during the dotcom boom of the 1990s.”
MethodSoftware architecture-based self-adaptation in robotics
Alberts, E., Gerostathopoulos, I., Malavolta, I., Hernández, C., & Lago, P. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112258 · Cited by 13
Estado da Arte sobre Afetividade na Formação de Grupos em Ambientes Colaborativos de Aprendizagem
Reis, R. C. D., Rodriguez, C. L., Lyra, K. T., Jaques, P. A., Bittencourt, I. I., & Isotani, S. (2015) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2015.23.03.113 · Cited by 18
Educating to achieve healthy open source ecosystems
Amorim, S. D. S., McGregor, J. D., de Almeida, E. S., & Chávez, C. (2018) · European Conference on Software Architecture · doi:10.1145/3241403.3241433 · Cited by 7
“To complement the research, we also performed a manual search in the papers collected by automated search undertaking a backward snowballing[23], searching their reference lists.”
MethodAgile Usage in Embedded Software Development in Safety Critical Domain–A Systematic Review
Demissie, S., Keenan, F., Özcan‐Top, Ö., & McCaffery, F. (2018) · Communications in computer and information science · doi:10.1007/978-3-030-00623-5_22 · Cited by 6
A Process for Tailoring Domain-Specific Enterprise Architecture Maturity Models
van Zwienen, M., Ruiz, M., van Steenbergen, M., et al. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-20618-5_14 · Cited by 6
Malapane, T. A. (2019) · Systems and Information Engineering Design Symposium · doi:10.1109/sieds.2019.8735644 · Cited by 8
How to develop accessible web interfaces for deaf people?
do Carmo, G. M., Paiva, D. M. B., & Cagnin, M. I. (2019) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3357155.3358437 · Cited by 6
“Como sugestões de trabalhos futuros espera-se estender o mape-amento sistemático realizado utilizando a técnica snowballing [47] e adicionando outros eventos importantes da área, como o Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia) e o Symposium on Engineering Interactive Computing…”
BackgroundDisclosure of Environmental Compliance Management on Corporate Websites
Thimm, H. H., & Rasmussen, K. B. (2019) · International Journal of Sustainable Entrepreneurship and Corporate Social Responsibility · doi:10.4018/ijsecsr.2020010103 · Cited by 6
Kudo, T. N., Bulcão-Neto, R. F., Vincenzi, A. M. R., & Macedo, A. A. (2019) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2019.458 · Cited by 6
“To check whether other essential studies on (Fabbri et al., 2013; Wohlin, 2014) this research exist, we also analyzed the list of bibliographic references as well as the citing papers of each one of these 10 studies.”
BackgroundMethodMapping the State of Security Standards Mappings
University of Innsbruck, D. O. C. S., Mussmann, A., Brunner, M., & Breu, R. (2020) · WI2020 Zentrale Tracks · doi:10.30844/wi_2020_l4-mussmann · Cited by 5
“To ensure the completeness of the identified resources, we performed one snowballing iteration [16].”
Methody Computación, D. D. S., Ponce-Corona, E., Sánchez, M. G., y Computación, D. D. S., Fajardo‐Delgado, D., y Computación, D. D. S., Acevedo-Juárez, B., y Exactas, D. D. C. N., De‐la‐Torre, M., Ingenierías, D. D. C. C. E., et al. (2020) · RISTI - Revista Ibérica de Sistemas e Tecnologias de Informação · doi:10.17013/risti.36.82-101 · Cited by 9
How do Business Factors Affect Developers in Mobile Software Ecosystems?
Steglich, C., Marczak, S. D. S., dos Santos, R. P., et al. (2020) · doi:10.1145/3411564.3411571 · Cited by 6
“…al. O conjunto de estudos identificados por meio deste processo foram complemen-tados utilizando a técnica de snowballing , apresentada por Wohlin [33]. snowballing O processo adotado para este estudo está demonstrado na Figura 1, cujas etapas são: (1) Definição das questões de pesquisa, (2)…”
BackgroundA systematic mapping study of performance analysis and modelling of cloud systems and applications
Odun-Ayo, I. A., Williams, T.-A., Odusami, M., & Yahaya, J. B. (2021) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v11i2.pp1839-1848 · Cited by 7
“Thereafter, a backward snowballing was carried out to complement the search process [31].”
MethodPatterns Related to Microservice Architecture: a Multivocal Literature Review
Valdivi, J. A., Lora-Gonzalez, A., Limón, X., Cortés-Verdín, K., & Ocharán-Hernández, J. O. (2021) · Proceedings of the Institute for System Programming of RAS · doi:10.15514/ispras-2021-33(1)-6 · Cited by 10
Wohlin, C., Papatheocharous, E., Carlson, J., Petersen, K., Alégroth, E., Axelsson, J., Badampudi, D., Borg, M., Cicchetti, A., Ciccozzi, F., et al. (2021) · Journal of Software Evolution and Process · doi:10.1002/smr.2345 · Cited by 13
“43 The main objective of the review was to identify factors that could influence the decision to choose among different component origins and solutions.”
Self-citationRecent Research on Phishing Detection Through Machine Learning Algorithm
Do, N. Q., Selamat, A., & Krejcar, O. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-79457-6_42 · Cited by 8
Computational Solutions for Human Falls Classification
Costa, E., Andrade, R. M. C., Rocha, L. S., Taramasco, C., & Ferreira, L. (2021) · IEEE Access · doi:10.1109/access.2021.3132796 · Cited by 7
“Moreover, we used the snowballing strategy [18] to increase the coverage of articles and possible inconsistency of the query string.”
MethodShepherd, J., Lourida, I., & Meertens, R. M. (2022) · BMC Emergency Medicine · doi:10.1186/s12873-022-00616-6 · Cited by 7
“A further search of the included articles reference lists and forward citation chasing was conducted [12, 21].”
BackgroundPatnaik, N., Dwyer, A. C., Hallett, J., & Rashid, A. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3561383 · Cited by 13
“We used a snowball-method [88] on the papers identified from the online search; and using forward and backward snowballing, we checked to see if there were other relevant papers.”
MethodSystematic Theory Mapping: Deciphering Causal Complexity of Brand Externalities
Padela, S. M. F., Wooliscroft, B., & Ganglmair‐Wooliscroft, A. (2023) · Journal of Macromarketing · doi:10.1177/02761467231157616 · Cited by 12
“This led to the snowballing search based on the guidelines by Wohlin (2014).”
MethodA systematic mapping of performance in distributed stream processing systems
Vogel, A., Henning, S., Ertl, O., & Rabiser, R. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00052 · Cited by 13
“In the last step, we conducted backward and forward snowballing [31], covering the control studies and most relevant primary studies found.”
MethodRequirements for adopting software process lines
Agh, H., García, F. O., Piattini, M. G., & Ramsin, R. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110546 · Cited by 5
“Backward snow-balling ( Wohlin, 2014 ) was also performed by examining the reference lists of the papers after the final filtering of the papers found by the automatic search.”
Text Mining Business Policy Documents: Applied Data Science in Finance
Spruit, M. R., & Ferati, D. (2020) · International Journal of Business Intelligence Research · doi:10.4018/ijbir.20200701.oa1 · Cited by 6
“We employ the snowballing approach as defined in (Wohlin, 2014) to determine to what extent these concepts have been addressed before.”
MethodVisual Analytics System for Energy Data in Smart Cities and Buildings
Alves, A. P., Milani, A. M. P., & Manssour, I. H. (2020) · IEEE International Smart Cities Conference (ISC2) · doi:10.1109/isc251055.2020.9239006 · Cited by 7
A Review on Ontology Modularization Techniques - A Multi-Dimensional Perspective
LeClair, A., Marinache, A., El-Ghalayini, H., MacCaull, W., & Khédri, R. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3152928 · Cited by 10
“An additional search referred to as backward snowball-ing [39] followed.”
MethodA Decision Support System for Blockchain Platform Selection
Verkleij, J. R. Q. (2018) · Utrecht University Repository (Utrecht University) · Cited by 6
Exploring Research Interest in Stack Overflow -- A Systematic Mapping Study and Quality EvaluationPreprint
Meldrum, S., Licorish, S. A., & Savarimuthu, B. T. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12282 · Cited by 13
“Checks were done using a snowballing approach, where the titles and abstract of all papers in the 14 proceedings were perused for potential relevance to our study [50].”
MethodNavigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Wijesinghe, S. M., & Mukaetova‐Ladinska, E. B. (2024) · OBM Geriatrics · doi:10.21926/obm.geriatr.2402279 · Cited by 5
A theory on human factors in DevOps adoption
Pérez-Sánchez, J., Rafi, S., de Gea, J. M. C., Nicolás, J., & Fernández‐Alemán, J. L. (2024) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103907 · Cited by 13
Machine Learning in Space: Surveying the Robustness of On-Board ML Models to Radiation
Lange, K., Fontana, F., Rossi, F., Varile, M., & Apruzzese, G. (2024) · IEEE Space Computing Conference (SCC) · doi:10.1109/scc61854.2024.00012 · Cited by 12
“We mostly rely on the snowball method [49].”
A Systematic Literature Review of Multi-Label Learning in Software Engineering
Hämäläinen, J., Das, T., & Mikkonen, T. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708532 · Cited by 10
“To expand the set of targeted articles with potentially more relevant studies, we applied snowballing [129] to the core set of papers by examining the references (backward snowballing) and identifying citing studies (forward snowballing).”
Calvén, A., Beery, T. H., Kristofers, H. Y., Johansson, M., Carlbäck, M., & Wendin, K. (2025) · Frontiers in Sustainable Tourism · doi:10.3389/frsut.2025.1529233 · Cited by 10
The technological landscape of collaborative model-driven software engineering
Choudhury, A., Malavolta, I., Ciccozzi, F., et al. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01274-5 · Cited by 5
“However, it is worth noting that in recent years, new technologies have emerged that are independent oflanguages, suchastheLanguageServerProtocol(LSP) 14 ., allowinguserstomaintaincontinuityandbuildexpertisewith asingletechnology.”
InfluentialRodriguez, S. D., Windl, M., Alt, F., & Marky, K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713968 · Cited by 6
“We filtered the publications for relevance.”
Moradzadeh, S., Zhang, Z., Gui, X., & Kou, Y. (2025) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-025-09519-z · Cited by 12
Gesso, C. D. (2025) · Management Research Review · doi:10.1108/mrr-09-2024-0714 · Cited by 11
From linear to loop: practical frameworks for achieving circularity in additive manufacturing
Tavares, T. M., Ganga, G. M. D., Filho, M. G., Osiro, L., & Tortorella, G. L. (2025) · Production Planning & Control · doi:10.1080/09537287.2025.2524559 · Cited by 9
“Additionally, four relevant papers were included using the backward snowballing method described by Wohlin (2014).”
Arno, P. (2025) · IEEE Access · doi:10.1109/access.2025.3585261 · Cited by 12
Open Source, Hidden Costs: A Systematic Literature Review on OSS License Management
Li, B., Liu, C., Fan, L., Chen, S., Zhang, Z., & Liu, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3586411 · Cited by 19
“To further understand the latest developments in OSS license management, we conducted an SLR on the subject and followed existing well-established guidelines [46], [47], [48], [49].”
Tiralti, A., & Rocchi, L. (2025) · Environment and Behavior · doi:10.1177/00139165251353473 · Cited by 7
“This process resulted in a sample of 147 papers plus 8 additional ones identified by “snowballing,” the process of searching the references and/or the citations of the articles to identify other relevant material (Wohlin, 2014), occurring in a final total of 155 selected articles.”
Lusi, J., Lorenz, B., & Pappel, I. (2025) · Journal of Open Innovation Technology Market and Complexity · doi:10.1016/j.joitmc.2025.100593 · Cited by 11
Matraf, M. S. B., Hashim, N. L., Hussain, A., Habbal, A., Hamouda, H. N., & Ghazali, O. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00180-6 · Cited by 6
“Additional papers were also identified using the reference lists of the existing articles, a method known as snowballing (Wohlin 2014).”
Characterizing time-critical internet of things
Leclerc, S., Bucaioni, A., & Ashjaei, M. (2025) · Internet of Things · doi:10.1016/j.iot.2025.101721 · Cited by 9
Irajpour, A., Baró, F., Edwards, M., Moosavi, S., Lacroix, É., & Llaguno-Munitxa, M. (2025) · Urban forestry & urban greening · doi:10.1016/j.ufug.2025.129040 · Cited by 7
Balancing Business, IT, and Human Capital: RPA Integration and Governance Dynamics
Brás, J. C., Pereira, R., Melo, M., Bianchi, I. S., & Ribeiro, R. (2025) · Information · doi:10.3390/info16090793 · Cited by 10
Collapse and imposition in the regulation of fire safety
Mohan, A. T., Seligmann, B. J., Coile, R. V., & Lange, D. (2025) · Safety Science · doi:10.1016/j.ssci.2025.106981 · Cited by 10
Consumer Repair Decision‐Making: A Systematic Literature Review and Future Research Agenda
Ozanne, L. K., & Madhuhansi, W. H. T. (2025) · International Journal of Consumer Studies · doi:10.1111/ijcs.70127 · Cited by 8
A survey of classification tasks and approaches for legal contracts
Singh, A., Joshi, A., Jiang, J., & Paik, H.-Y. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11359-8 · Cited by 19
Modular Monolith Architecture in Cloud Environments: A Systematic Literature Review
Al-Qora’n, L. F., & Ahmad, A. A.-S. (2025) · Future Internet · doi:10.3390/fi17110496 · Cited by 13
“In addition, backward snowballing [27] was conducted to search for any relevant references in the included papers that were not retrieved using our search string.”
InfluentialMcKay, L., McAloone, T. C., & Pigosso, D. C. A. (2025) · Sustainable Production and Consumption · doi:10.1016/j.spc.2025.10.019 · Cited by 7
Personalized Music Education: A Systematic Review of AI Generation Methods
Carnovalini, F., Santo, L. E., & Wiggins, G. A. (2025) · IEEE Access · doi:10.1109/access.2025.3639898 · Cited by 9
Accessibility by Design: A Systematic Review of Inclusive E-Book Standards, Tools, and Practices
Silva, L. C. E., Pimentel, B. A., da Costa, B. J. D., Filho, R. S. E., Sousa, L., da Cruz, N. J. T., & Silva, R. D. A. (2025) · Sustainability · doi:10.3390/su172411173 · Cited by 6
Integrating AI into healthcare systems: A multivocal literature review
Mallardi, G., Calefato, F., Quaranta, L., & Lanubile, F. (2025) · Smart Health · doi:10.1016/j.smhl.2025.100631 · Cited by 9
Ferino, S., Hoda, R., Grundy, J., & Treude, C. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3800580 · Cited by 42
Security Debt: Characteristics, Product Life-Cycle Integration and Items
Martinez, J., Quintano, N., Ruíz, A., Santamaría, I., de Soria, I. M., & Arias, J. (2021) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1109/techdebt52882.2021.00009 · Cited by 8
“This set was used for a snowballing approach [26] where potentially related”
MethodGreen, J., & Stracener, J. T. (2022) · Systems · doi:10.3390/systems10050169 · Cited by 5
Vivanco‐Suarez, J., Sioutas, G. S., Matache, I., Muhammad, N., Salem, M. M., Kandregula, S., Jankowitz, B. T., Burkhardt, J., & Srinivasan, V. M. (2023) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2022-020035 · Cited by 5
Systematic Literature Review of Machine Learning in Virtual Reality and Augmented Reality
Sindu, I. G. P., Hartati, R. S., Sudarma, M., & Gunantara, N. (2023) · Jurnal Nasional Pendidikan Teknik Informatika (JANAPATI) · doi:10.23887/janapati.v12i1.60126 · Cited by 14
Falduti, M., & Tessaris, S. (2023) · Digital Threats Research and Practice · doi:10.1145/3608483 · Cited by 11
“In order to assess completeness in collecting the studies, we followed the snowball strategy presented in [74], adapted on the authors keywords as depicted in Fig.”
MethodKnowledge sharing in open-source software development communities: a review and synthesis
Okong’o, W., & Ndiege, J. R. (2023) · VINE Journal of Information and Knowledge Management Systems · doi:10.1108/vjikms-10-2022-0339 · Cited by 5
Towards Scalable and Privacy-Enhanced On-Street Parking Management: A Roadmap for Future Inquiry
Alahmadi, S., Hakeem, A., Attiah, A., Alghamdi, B. M., & Mohaisen, L. (2023) · Electronics · doi:10.3390/electronics12194160 · Cited by 5
A systematic review of mechanistic models of riverine macrophyte growth
Dietterich, L. H., Ortiz-Rosa, S., Charbonneau, B. R., & McKay, S. K. (2023) · Aquatic Botany · doi:10.1016/j.aquabot.2023.103724 · Cited by 6
Towards a Catalog of Refactorings for Elixir
Vegi, L. F. D. M., & Valente, M. T. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00045 · Cited by 7
“3) Snowballing: In order to add new studies to our knowledge base, we used a snowballing process based on the recommendations made by Wohlin [15].”
MethodLiu, W., Chan, A. P. C., Chan, M. W., Darko, A., & Oppong, G. D. (2024) · Engineering Construction & Architectural Management · doi:10.1108/ecam-10-2023-1060 · Cited by 11
When rationality meets intuition: A research agenda for software design decision‐making
Pretorius, C., Razavian, M., Eling, K., & Langerak, F. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2664 · Cited by 7
“Using a systematic literature review, 32,33 the current study identifies and categorizes existing research into an overview based on the when and how of using and priming intuition and rationality in software design decision-making.”
Vitkos, E. N., Kounatidou, N. E., Konsolaki, E., et al. (2024) · Journal of Stomatology Oral and Maxillofacial Surgery · doi:10.1016/j.jormas.2024.101845 · Cited by 5
A systematic mapping on software testing for blockchains
Elakaş, A., Sözer, H., Şafak, I., & Kalkan, K. (2024) · Cluster Computing · doi:10.1007/s10586-024-04421-7 · Cited by 6
“Snowballing: We performed snowballing [19] on the papers already included in our list after Phase 5.”
Horber, D., Pickel, J., Goetz, S., & Wartzack, S. J. (2024) · IEEE Open Journal of Systems Engineering · doi:10.1109/ojse.2024.3434310 · Cited by 7
“In order to improve re-producibility, the study and its documentation focuses on the results that can be retrieved with the search string from sco-pus, which are provided in the replication package [46] within the supplementary material of this contribution.”
BackgroundMethod
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.