Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 1,901–2,000 of 3,934
Comparison of Agile Maturity Models
Schmitt, A., Theobald, S., & Diebold, P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-35333-9_52 · Cited by 8
Haindl, P., & Plösch, R. (2020) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa51224.2020.00071 · Cited by 12
“We followed the process as outlined by Wohlin [26] and selected the 115 papers remaining after exclusion based on abstract, introduction, or conclusion (cf. Figure 1) as the tentative start set for the snowballing procedure.”
MethodInfluentialAchieving Agility in IT Project Portfolios – A Systematic Literature Review
Chakko, J. P., Huygh, T., & Haes, S. D. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-67084-9_5 · Cited by 14
TestDCat 3.0: catalog of test debt subtypes and management activities
Aragão, B. S., Andrade, R. M. C., Santos, I. S., Castro, R. N. S., Lelli, V., & Darin, T. (2021) · Software Quality Journal · doi:10.1007/s11219-020-09533-y · Cited by 8
“Based on the principle of Snowballing Wohlin (2014), we also performed a search for studies that cited the set of previously selected papers.”
MethodReshaping the Concepts of Job Enrichment and Job Enlargement: The Impacts of Lean and Industry 4.0
Lagorio, A., Cimini, C., & Gaiardelli, P. (2021) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-85874-2_79 · Cited by 9
Barriers of managing cloud outsource software development projects: a multivocal study
Akbar, M. A., Mahmood, S., Meshram, C. Y., Alsanad, A., Gumaei, A., & AlQahtani, S. A. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11245-9 · Cited by 15
“The backward snowballing approach refer to the studies cited in the paper and forward snowballing refer to the studies by whom the paper is cited [34, 38].”
MethodAn Overview of Ontologies and Tool Support for COVID-19 Analytics
Ahmad, A., Bandara, M., Fahmideh, M., Proper, H. A., Guizzardi, G., & Soar, J. (2021) · IEEE International Enterprise Distributed Object Computing Conference Workshops (EDOCW) · doi:10.1109/edocw52865.2021.00026 · Cited by 8
Big Data Testing Techniques: Taxonomy, Challenges and Future Trends
Arshad, I., Alsamhi, S. H., & Afzal, W. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.030266 · Cited by 13
A Systematic Literature Review on Software Maintenance for Cyber-Physical Systems
Khezami, N., Kessentini, M., & Ferreira, T. (2021) · IEEE Access · doi:10.1109/access.2021.3126681 · Cited by 8
Quality Models and Quality Attributes for Open Educational Resources: A Systematic Mapping
de Oliveira, M. M., Paschoal, L. N., & Barbosa, E. F. (2021) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie49875.2021.9637309 · Cited by 8
“(4)Snowballing refers to the use of a study’s reference list (backward snowballing) or the list of studies that cited that study (forward snowballing) to identify additional relevant studies [18].”
BackgroundTasoudis, P., Magouliotis, D. E., Varvoglis, D. N., et al. (2022) · General Thoracic and Cardiovascular Surgery · doi:10.1007/s11748-022-01792-9 · Cited by 8
“The references of the relevant reviews or meta-analysis were hand-searched for potentially eligible studies using the snowball methodology [15].”
MethodTsiouvalas, A., Stoeva, G., & Raspotnik, A. (2022) · Sustainability · doi:10.3390/su14074115 · Cited by 13
Estudantes com Deficiência Visual em Computação: participação, perspectivas e desafios enfrentados
Alves, L. F., Rocha, L., Pereira, C. P., et al. (2022) · doi:10.5753/educomp.2022.19200 · Cited by 8
“O processo de busca por trabalhos relacionados foi feito nas bases IEEE Xplore, ACM Digital Library e Google Schoolar, a partir de termos de busca tanto em inglês quanto português, também utilizanda a técnica Snowballing [39] a partir de artigos encontrados nas bases de pesquisa.”
BackgroundThe Use of Open Government Data to Create Social Value
Reyes, M. E. L., & Magnussen, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15086-9_16 · Cited by 8
Seth, M., Jalo, H., Högstedt, Å., Medin, O., Björner, U., Sjöqvist, B. A., & Candefjord, S. (2022) · JMIR Research Protocols · doi:10.2196/40243 · Cited by 10
“In addition, a snowball approach described by Wohlin [60] was used to further screen for new articles.”
“To Clean Code or Not to Clean Code” A Survey Among Practitioners
Ljung, K., & González‐Huerta, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21388-5_21 · Cited by 21
Toward Stress Detection During Gameplay: A Survey
Wijayarathna, C., & Lakshika, E. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3216404 · Cited by 11
“To achieve this, we started to search literature using backward snowballing [18] starting from the initial set of papers where we identified relevant research by looking into the references of the papers.”
MethodBorangiu, T., Trentesaux, D., & Leitão, P. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-24291-5 · Cited by 12
Understanding the customer experience in human-computer interaction: a systematic literature review
Quiñones, D., & Rojas, L. F. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1219 · Cited by 22
“Articles were not excluded if an author is not known for publishing in the CX or HCI area, but a article may be more likely to be included if the author regularly publishes in the CX or HCI area (Wohlin, 2014).”
BackgroundMethodChrzanowski, M., & Zawada, P. (2023) · Energies · doi:10.3390/en16052084 · Cited by 8
Dürr, K., & Lichtenthäler, R. (2022) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1109/ucc56403.2022.00053 · Cited by 10
“We filtered the results incrementally based on titles, abstracts and full-texts and did one iteration of the snowballing approach [28].”
MethodAdvancing database security: a comprehensive systematic mapping study of potential challenges
Iqbal, A., Khan, S. U., Niazi, M., Humayun, M., Sama, N. U., Khan, A. A., & Ahmad, A. (2023) · Wireless Networks · doi:10.1007/s11276-023-03436-z · Cited by 15
“We used the snowballing approach [42–44] in addition to the previous inclusion/exclusion criteria for our concluding decision.”
García-Jaén, M., Sebastiá-Amat, S., Sanchís-Soler, G., & Cortell-Tormo, J. M. (2024) · Healthcare · doi:10.3390/healthcare12070758 · Cited by 16
“Searches of websites considered eligible in terms of both organization and date were also conducted [70,72].”
BackgroundVargas, D., & Fontoura, L. M. (2024) · International journal of information systems and project management · doi:10.12821/ijispm120103 · Cited by 7
A systematic mapping study on requirements scoping
Wnuk, K., & Kollu, R. K. (2016) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2915970.2915985 · Cited by 10
“Moreover, we followed the snowballing guidelines suggested by Wohlin [45], the mapping study guidelines suggested by Petersen et al.”
BackgroundMethodInfluentialEffort Estimation for ERP Projects — A Systematic Review
Ömüral, N. K., & Demirörs, O. (2017) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2017.68 · Cited by 9
“Characteristic “research approach” is used relying on the work presented in [47]; corresponding categories are explained in detail in Appendix A.”
BackgroundMethodAgile and lean metrics associated with requirements engineering
Wnuk, K., & Maddila, K. C. (2017) · Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement (IWSM-MENSURA) · doi:10.1145/3143434.3143437 · Cited by 13
Haegemans, T., Snoeck, M., & Lemahieu, W. (2018) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2018.05.009 · Cited by 12
Effort Estimation in Agile Software Development: an Updated Review
Filho, E. D., Perkusich, M., Dilorenzo, E., et al. (2018) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2018-003 · Cited by 8
Wouters, J., Ritmeester, J. R., Carlsen, A. W., Jansen, S., & Wnuk, K. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-33742-1_4 · Cited by 10
“We also applied the snowballing methodology [28] to identify other relevant papers.”
MethodOLAP parallel query processing in clouds with C‐ParGRES
Ribeiro, M. W. M., Lima, A. A. B., & de Oliveira, D. (2019) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5590 · Cited by 8
Current State and Next Steps on Automated Hints for Students Learning to Code
Toll, D., Wingkvist, A., & Ericsson, M. (2020) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie44824.2020.9274053 · Cited by 8
The screening phase in systematic reviews: Can we speed up the process?
Rožanc, I., & Mernik, M. (2021) · Advances in computers · doi:10.1016/bs.adcom.2021.01.006 · Cited by 23
A Literature Review of Machine Learning Techniques for Cybersecurity in Data Centers
Roponena, E., Kampars, J., Gailitis, A., & Strods, J. (2021) · International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS) · doi:10.1109/itms52826.2021.9615321 · Cited by 10
Comprehending the use of intelligent techniques to support technical debt management
Albuquerque, D. W., Guimarães, E. T., Tonin, G., Perkusich, M., de Almeida, H. O., & Perkusich, Â. (2022) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3524843.3528097 · Cited by 14
“Whilethisstrategysignificantlyincreasedthesearcheffort[33],it enabledustofindacomprehensivesetoftherelevantstudy.”
A Systematic Literature Review of Social Network Systems for Older Adults
Ahmad, B., Richardson, I., & Beecham, S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-69926-4_38 · Cited by 10
A Graph-Based Tag Recommendation for Just Abstracted Scientific Articles Tagging
Boughareb, D., Khobizi, A., Boughareb, R., Farah, N., & Séridi, H. (2020) · International Journal of Cooperative Information Systems · doi:10.1142/s0218843020500045 · Cited by 8
Architectural Tactics for Big Data Cybersecurity Analytic Systems: A ReviewPreprint
Ullah, F., & Babar, M. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03178 · Cited by 8
“In order to ensure the identification and selection of as many relevant papers as possible, we complemented the automatic search with snowballing [16, 17].”
MethodA Systematic Mapping Study on Testing of Machine Learning ProgramsPreprint
Sherin, S., Khan, M. U., & Iqbal, M. Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.09427 · Cited by 25
Evolution of Low-Code Platforms
Overeem, M. (2022) · Utrecht University Repository (Utrecht University) · doi:10.33540/1197 · Cited by 15
Violence Against Parents by Adult Children: A Systematic Literature Review of Empirical Studies
Klun, M., Ručman, A. B., & Frangež, D. (2024) · Trauma Violence & Abuse · doi:10.1177/15248380241280955 · Cited by 12
“Based on these papers, a literature review was performed using the snowball method according to Wohlin (2014).”
Thalpage, N. S., & Nisansala, T. A. D. (2023) · Springer eBooks · doi:10.1007/978-3-031-34006-2_4 · Cited by 8
Circular Economy Implementation from the Perspective of Benefits and Barriers
Zemanová, V. (2023) · Liberec Economic Forum (LEF) · doi:10.15240/tul/009/lef-2023-31 · Cited by 12
Mapping Renewable Energy among Antarctic Research Stations
de Witt, M., Chung, C., & Lee, J. (2024) · Sustainability · doi:10.3390/su16010426 · Cited by 7
de Sousa, L. J., Martins, J. P., Sanhudo, L., & Baptista, J. S. (2024) · Construction Innovation · doi:10.1108/ci-12-2022-0315 · Cited by 17
Rodríguez, G., & Torres, J. (2024) · Annals of Telecommunications · doi:10.1007/s12243-024-01022-8 · Cited by 10
“As mentioned in [66], the snowball technique uses an article’s reference list and citations to identify additional articles.”
Akresh, M. E., McInvale, S. L., & King, D. I. (2024) · Journal of Field Ornithology · doi:10.5751/jfo-00481-950212 · Cited by 12
Tasoudis, P., Loufopoulos, G., Manaki, V., et al. (2024) · Lung Cancer · doi:10.1016/j.lungcan.2024.107929 · Cited by 7
Frankford, E., Höhn, I., Sauerwein, C., & Breu, R. (2024) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10662990 · Cited by 18
“After searching the classical databases Google Scholar, a round of snowballing was conducted [6].”
MethodAI Simulation by Digital Twins: Systematic Survey of the State of the Art and a Reference Framework
Liu, X., & Dávid, I. (2024) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3652620.3688253 · Cited by 12
“In backward snowballing, we follow Wohlin [79] and mark potentially relevant references as we examine studies in the data extraction phase.”
Blockchain for Securing CI/CD Pipeline: A Review on Tools, Frameworks, and Challenges
Saleh, S. M., Madhavji, N. H., & Steinbacher, J. (2024) · Conference on Cloud and Internet of Things (CIoT) · doi:10.1109/ciot63799.2024.10757084 · Cited by 11
“To maintain the Quasi-Gold standard [14], we ran a manual search to check other evidence resources, such as the citations of the primary studies and review articles—this technique is known as Snowballing [15, 19].”
Design science in operations management: A review and synthesis of the literature
Gauss, L., Lacerda, D. P., Siluk, J. C. M., & Romme, A. G. L. (2024) · International Journal of Management Reviews · doi:10.1111/ijmr.12390 · Cited by 18
Analogy competence for science teachers
Eriksson, S., Gericke, N., & Thörne, K. E. (2024) · Studies in Science Education · doi:10.1080/03057267.2024.2434797 · Cited by 18
“In this way, 30 additional studies were identified in line with a backward and forward snowballing method (Wohlin, 2014).”
Research on absorptive capacity in the green context: a bibliometric and visualization analysis
Ni, C., & Abdullah, N. L. (2025) · Cogent Business & Management · doi:10.1080/23311975.2024.2435602 · Cited by 24
“Using specific and precise keywords helps avoid retrieving irrelevant articles (Wohlin, 2014). consequently, a search string was developed to combine keywords with Boolean operators: (‘absorptive capacity’) anD (‘green’ OR ‘environmental*’ OR ‘low carbon’ OR ‘eco-friendly’ OR ‘eco-innovation’). the…”
Mining software repositories for software architecture — A systematic mapping study
Soliman, M., Albonico, M., Malavolta, I., & Wortmann, A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107677 · Cited by 18
Requirements extraction from model-based systems engineering: A systematic literature review
Santos, J., Martins, L. E. G., & Molléri, J. S. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112407 · Cited by 14
Miao, T., Shaafi, A. I., & Song, E.-J. (2025) · Electronics · doi:10.3390/electronics14071397 · Cited by 21
Vilaplana, J. A. L., Yang, G., Ackom, E. K., & Ackom, E. (2025) · Energy Research & Social Science · doi:10.1016/j.erss.2025.104052 · Cited by 15
Ghaffar, A., Din, I. U., Tariq, A., et al. (2025) · Review of Education · doi:10.1002/rev3.70071 · Cited by 7
Verifiability for privacy-preserving computing on distributed data — a survey
Bontekoe, T., Karastoyanova, D., & Türkmen, F. (2025) · International Journal of Information Security · doi:10.1007/s10207-025-01047-7 · Cited by 33
QIoT: IoT Architectures in Quantum Computing Era
Alipour, M. (2025) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c65153.2025.00044 · Cited by 10
“Afterward, inclusion and exclusion criteria (see Table I) were applied (initially on abstracts and further on full text) to filter out irrelevant studies (17 papers), followed by one iteration of forward and backward snowballing [20] without any publisher limitation.”
Human-Machine Teaming and Team Effectiveness in AI Tools for Software Engineering
Rauf, I., Sharp, H., López, T., & Wermelinger, M. (2025) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase66643.2025.00017 · Cited by 19
“We identified seven new studies through forward snowballing and backward snowballing [40]; reading their abstracts and removing duplicates left five new studies.”
The Effects of Climate Change on Mesocarnivores: A Global Review and Meta‐Analysis
Valdez, V., Ferreras, P., & Rosalino, L. M. (2025) · Global Change Biology · doi:10.1111/gcb.70302 · Cited by 10
“We complemented our literature search using a snow-balling approach (Wohlin 2014), where we included relevant studies cited in the articles selected by our string.”
How small businesses build their brands in a digital world: a systematic review
Fluhrer, P., & Brahm, T. (2025) · Review of Managerial Science · doi:10.1007/s11846-025-00931-9 · Cited by 46
A STAMP-Informed framework for classifying interorganizational risk management challenges in ports
Meléndez, E., & Goerlandt, F. (2025) · Safety Science · doi:10.1016/j.ssci.2025.107000 · Cited by 12
Evaluating the quality of GenAI applications in software engineering: a multi-case study
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10759-2 · Cited by 12
“The review study followed Wohlin (2014 ) guidelines and identified peer-reviewed metrics used to eval - uate AI-generated outputs in software engineering contexts.”
Paes, V. M., Ting, A. S. J., Paes, M. V. I., Masters, J., Graham, S., & Costa, M. L. (2026) · The Bone & Joint Journal · doi:10.1302/0301-620x.108b1.bjj-2025-0035.r1 · Cited by 12
A systematic review of multimodal learning analytics in computer-supported collaborative learning
Ouyang, F., & Bai, X. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105574 · Cited by 15
The impact of Use Cases in real-world software development projects: A systematic mapping study
Barros‐Justo, J. L., Benitti, F. B. V., & Tiwari, S. (2019) · Computer Standards & Interfaces · doi:10.1016/j.csi.2019.103362 · Cited by 9
“We conducted a snowballing search using the papers selected from the automated search strategy as the initial set of seeds, following the guidelines proposed in [47].”
MethodDigital Tools to Promote Healthy Eating for Working-Age Individuals: A Scoping Review
Pan, S., Ren, X., Vos, S. B., & Brombacher, A. (2021) · International Symposium of Chinese CHI (Chinese CHI) · doi:10.1145/3490355.3490356 · Cited by 13
“Third, a snowballing procedure in the reference lists of the selected papers and the citations was manually performed to identify additional papers [52].”
MethodVisual Exploration of Literature with Argo Scholar
Li, K., Yang, H., Montoya, E., Upadhayay, A., Zhou, Z., Saad-Falcon, J., & Chau, D. H. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557177 · Cited by 8
“Academic search engines [2, 19] are common starting point [6, 12]; however, keyword searching does not provide users with all relevant papers [26], so incremental exploration, a common and effective method where users iterate from familiar (identified via search engines) to novel works while filtering based on paper a…”
BackgroundBringing Computational Thinking to Life Through Play
Kaup, C. F., Møller, A. K., & Brooks, E. (2023) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-31392-9_9 · Cited by 9
The dynamic versus the stable team: The unspoken question in large‐scale agile development
Ståhl, D. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2589 · Cited by 13
International development project success: A literature review
Chasanah, N., Gunawan, I., & Baroudi, B. (2023) · Journal of International Development · doi:10.1002/jid.3809 · Cited by 14
Zhang, Y., & Chen, K. (2023) · Journal of Civil Engineering and Management · doi:10.3846/jcem.2023.19574 · Cited by 7
Eramo, R., Tucci, M., Pompeo, D. D., Cortellessa, V., Marco, A. D., & Taibi, D. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111833 · Cited by 17
“We performed the snowballing process [35], by considering all the references presented in the retrieved papers and evaluating all the papers referencing the retrieved papers, which resulted in one additional relevant paper.”
BackgroundMethodInfluentialArdebili, A. A., Longo, A., & Ficarella, A. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386762 · Cited by 9
“The Third stage of the current review is conducting a systematic snowballing. the outline of the steps follows the process introduced by Wholin in 2014 to identify the relevant research papers from a selected set of papers [16].”
BackgroundChen, X., Usman, M., & Badampudi, D. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107451 · Cited by 21
Olumekor, M. O. (2024) · Management Review Quarterly · doi:10.1007/s11301-024-00433-0 · Cited by 12
“Forward and backward snowballing method (Wohlin 2014), and manual screening of 6 selected journals with broad relevance to PSE ( n = 19).”
A Theoretical Framework to Promote LCA in the Construction Industry of Saudi Arabia
Shaukat, M. M., Luqman, M., Asif, M., Shuja, S. Z., & Qannan, M. (2024) · Sustainability · doi:10.3390/su16093778 · Cited by 8
Uncovering rebound effects of sufficiency-oriented product-service systems: a systematic review
Andrew, E. M., van den Bergh, J. C. J. M., & Pigosso, D. C. A. (2024) · Proceedings of the Design Society · doi:10.1017/pds.2024.121 · Cited by 8
“Furthermore, backward snowballing (Wohlin, 2014) was applied, involving the cross-referencing of papers in the review, adding 70 publications.”
MethodA review of walk-and-talk conversations with children as a research method
Hammarsten, M. (2024) · Journal of Adventure Education & Outdoor Learning · doi:10.1080/14729679.2024.2404656 · Cited by 15
“In two cases, a forward snowball search (Wohlin, 2014) was also conducted, to look for studies that referred to any of the studies already found.”
An Empirical Study on Python Library Dependency and Conflict Issues
Jia, X., Zhou, Y., Hussain, Y., et al. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00057 · Cited by 7
“New relevant papers are found by collecting papers cited by the original paper and papers that cite it iteratively [32].”
BackgroundAkbar, M. A., Naseem, A., Zaman, U. K. U., & Petronijević, J. (2024) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-024-14601-9 · Cited by 8
“…(published since 2018) including the one added through forward snowball method (to search articles by their citations) in the light of guidance by [90] such as a relevant paper published in 2020 by Yousefi et al. [63] was observed in the citations of an article published in 2018 by Baghery et al.…”
Matzen, A., Rüffer, A., Byllemos, M., Heine, O., Παπαϊωάννου, Μ., Choudhary, G., & Dragoni, N. (2025) · Applied Sciences · doi:10.3390/app15084414 · Cited by 17
IMPACT ON DESIGN WHEN INTRODUCING ADDITIVE MANUFACTURING IN SPACE APPLICATIONS
Borgue, O., Panarotto, M., & Isaksson, O. (2018) · Proceedings of the ... International Design Conference/Design ... · doi:10.21278/idc.2018.0412 · Cited by 7
“For both searches, redundant items were removed and the remaining list was complemented with other entries implementing a procedure of backward snowballing (Wohlin, 2014) with the purpose of reaching”
BackgroundMethodAnalyzing Families of Experiments in SE: A Systematic Mapping Study
Santos, A., Gómez, O. S., & Juristo, N. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2864633 · Cited by 47
“, a procedure where relevant studies are gathered from the primary studies’ reference lists [16]).”
BackgroundMethodAnalysing the Evolution of Exception Handling Anti-Patterns in Large-Scale Projects
de Sousa, D. B. C., Maia, P. H. M., Rocha, L. S., et al. (2018) · doi:10.1145/3267183.3267191 · Cited by 7
“They have been selected using the systematic search strategy snow-balling [26].”
MethodQuantitative assessment of service delivery process: application of hybrid simulation modelling
Rondini, A., Pezzotta, G., & Cavalieri, S. (2018) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2018.08.454 · Cited by 7
O prontuário eletrônico do paciente em escala nacional
Galvão, M. C. B., & Ricarte, I. L. M. (2019) · InCID Revista de Ciência da Informação e Documentação · doi:10.11606/issn.2178-2075.v10i1p244-264 · Cited by 7
“Aos resultados selecionados foram adicionados artigos que citavam os artigos selecionados, aplicando a técnica de snowballing (WOHLIN, 2014).”
BackgroundJäderberg, L., Goss, S., & McBeath, A. G. (2019) · Counselling and Psychotherapy Research · doi:10.1002/capr.12279 · Cited by 7
Wijayarathna, C., Grobler, M. M., & Arachchilage, N. A. G. (2019) · Behaviour and Information Technology · doi:10.1080/0144929x.2019.1705393 · Cited by 7
“Furthermore, while reading the papers for data extraction (data extraction is described later in this section), we searched for papers that use the CDF-based questionnaire by looking at references of reviewed papers (backward snowballing Wohlin 2014).”
MethodTrisomy 8 mosaicism in the placenta: A Danish cohort study of 37 cases and a literature review
Thomsen, S. H., Lund, I. C. B., Fagerberg, C. R., et al. (2020) · Prenatal Diagnosis · doi:10.1002/pd.5875 · Cited by 7
“3 | Literature review A literature review was conducted by a systematic search in PubMed and inclusion of cross references.(23) The systematic search in PubMed was conducted using the search terms “Chromosome 8, mosaic trisomy”[Supplementary Concept], “Mosaicism”[MeSH], “Prenatal Diagnosis”[MeSH], “Chromosome 8, triso…”
MethodNazir, M. (2020) · Lahore Garrison University Research Journal of Computer Science and Information Technology · doi:10.54692/lgurjcsit.2020.0404105 · Cited by 8
“Backward Snowballing: Cited by the more other research studies and part of references [4].”
BackgroundA Comprehensive Framework for Analyzing IoT Platforms: A Smart City Industrial Experience
Fahmideh, M., Yan, J., Shen, J., Mougouei, D., Zhai, Y., & Ahmad, A. (2021) · Smart Cities · doi:10.3390/smartcities4020031 · Cited by 13
“In adherence to MC9 (comprehensiveness), we used the snowballing technique [17] into the review process to include as many relevant papers as possible.”
MethodEvaluation of Non-Functional Requirements for IoT Applications
Paiva, J. O. V., Andrade, R. M. C., & Carvalho, R. M. (2021) · International Conference on Enterprise Information Systems · doi:10.5220/0010461901110119 · Cited by 7
“com based on a set of papers that cite the paper being examined (Wohlin, 2014).”
BackgroundMethodInfluentialIoT Security-Quality-Metrics Method and Its Conformity with Emerging Guidelines
Kosuke, I., Morisaki, S., & Goto, A. (2021) · IoT · doi:10.3390/iot2040038 · Cited by 7
“We conducted a systematic literature review [36] using a combination of keywords such as “IoT”, “security”, and “quality metrics” to find related work.”
MethodInfluentialKaiser, I., Diehl, K., Heppt, M. V., Mathes, S., Pfahlberg, A. B., Steeb, T., Uter, W. J. C., & Gefeller, O. (2022) · Healthcare · doi:10.3390/healthcare10020238 · Cited by 8
“Forward snowballing is an efficient search strategy that explores citations to specific reference papers and thus looks forward in time when performing a search among citations [14].”
BackgroundAli, B., & Dahlhaus, P. G. (2022) · Sustainability · doi:10.3390/su14063185 · Cited by 9
Franceschi, F. F., Clavijo, L. T. V., Sanches-Pereira, A., Cherni, J. A., & Gómez, M. F. (2022) · Clean Technologies and Environmental Policy · doi:10.1007/s10098-022-02333-x · Cited by 12
“Finally, the backward snowball technique was applied (Wohlin 2014), and 9 more studies were included.”
Method
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.