Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 801–900 of 3,934
Algorithmic and Ethical Aspects of Recommender Systems in e-Commerce
Paraschakis, D. (2018) · doi:10.24834/2043/24268 · Cited by 35
Loyalty and public transit: a quantitative systematic review of the literature
Carvalho, T., Romano, C. A., & Gadda, T. M. C. (2021) · Transport Reviews · doi:10.1080/01441647.2021.1991032 · Cited by 40
“As a complementary measure and to ensure that all relevant literature was assessed, we also applied backward snowballing (Wohlin, 2014).”
BackgroundSoftware defect prediction using hybrid techniques: a systematic literature review
Malhotra, R., Chawla, S., & Sharma, A. (2023) · Soft Computing · doi:10.1007/s00500-022-07738-w · Cited by 46
“We performed the snowballing procedure by scanning the references of the chosen studies and including them in our studies (Wohlin 2014).”
MethodGuzzo, D., Walrave, B., & Pigosso, D. C. A. (2023) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2023.137003 · Cited by 42
A Descriptive Analysis of a Formative Decade of Research in Affective Haptic System Design
Vyas, P., Desai, U., Yamakawa, K., & MacLean, K. E. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580735 · Cited by 50
Aghajani, M., Ruge, G., & Jugdev, K. (2023) · Project Management Journal · doi:10.1177/87569728231172668 · Cited by 56
“In parallel to the full-text fi ltering of articles (see Figure 1), we used Google Scholar to identify additional relevant articles from the references of the retrieved articles, following the Wohlin (2014) recommendations for snowballing.”
MethodSchreiter, S., Friedrich, A., Fuhr, H., Malone, S., Brünken, R., Kühn, J., & Vogel, M. (2023) · ZDM · doi:10.1007/s11858-023-01531-1 · Cited by 37
“In a final step, forward snowball searches (cf. Wohlin, 2014) were performed for each included study, adding another 11 studies.”
Lv, J.-L., Wei, Y., Sun, J., Shi, Y., Liu, F.-H., Sun, M.-H., Chang, Q., Wu, Q., & Zhao, Y. (2024) · Frontiers in Nutrition · doi:10.3389/fnut.2024.1306310 · Cited by 52
“Furthermore, included studies were backward snowballed manually and forward snowballed using the Web of Science citation tracking feature to identify additional eligible studies (27, 28).”
MethodWairimu, S., Iwaya, L. H., Fritsch, L., & Lindskog, S. (2024) · IEEE Access · doi:10.1109/access.2024.3360864 · Cited by 52
“Additionally, 5 studies were retrieved through forward and backward snowballing [44], resulting in a total of 39 studies relevant to this SLR. Bibliographies of the final results were exported to Zotero to share all included studies among authors.”
MethodZaguir, N. A., de Magalhães, G. H., & Spínola, M. D. M. (2024) · IEEE Access · doi:10.1109/access.2024.3406724 · Cited by 73
“Wohlin [30] referred to this sampling technique as ‘‘snowballing’’ and presented evidence of its successful application in replicating a published study, under-scoring the reliability of this approach.”
BackgroundAcceptance of or resistance to facial recognition payment: A systematic review
Yu, T., Wang, C., Bian, Q., & Teoh, A. P. (2024) · Journal of Consumer Behaviour · doi:10.1002/cb.2385 · Cited by 39
Network Digital Twin Toward Networking, Telecommunications, and Traffic Engineering: A Survey
Poorzare, R., Kanellopoulos, D. N., Sharma, V. K., Dalapati, P., & Waldhorst, O. P. (2025) · IEEE Access · doi:10.1109/access.2025.3531947 · Cited by 62
Gyevnár, B., & Kasirzadeh, A. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01020-y · Cited by 88
The Evolution of Reinforcement Learning in Quantitative Finance: A Survey
Pippas, N., Ludvig, E. A., & Turkay, Ç. (2025) · ACM Computing Surveys · doi:10.1145/3733714 · Cited by 74
Delgado, L. M., Souza, V. D. S., Pompeu, B. F., Ogawa, T. D. M., Oliveira, H. P., Valladão, V. D. C. S., & Marques, F. A. (2025) · Obesity Surgery · doi:10.1007/s11695-025-08044-8 · Cited by 40
Human Aspects in Software Architecture Decision Making: A Literature Review
Tang, A., Razavian, M., Paech, B., & Hesse, T.-M. (2017) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa.2017.15 · Cited by 49
“With the results from our search in the primary sources and the known papers (Step 4), we used a snowballing technique (Step 5) [15] to find relevant papers from citations.”
MethodSabir, F., Palma, F., Rasool, G., Guéhéneuc, Y., & Moha, N. (2018) · Software Practice and Experience · doi:10.1002/spe.2639 · Cited by 57
Meyer, M. M., Glas, A. H., & Eßig, M. (2020) · Management Review Quarterly · doi:10.1007/s11301-020-00198-2 · Cited by 42
“Based on the findings of the literature research, a citation search (Wohlin 2014) was conducted to expand the pool of literature based on a preselected and included set of publications within the literature review.”
BackgroundOn the Study of Microservices Antipatterns
Tighilt, R., Abdellatif, M., Moha, N., et al. (2020) · European Conference on Pattern Languages of Programs · doi:10.1145/3424771.3424812 · Cited by 35
“To minimize this threat, we (1) included in our search query the most important keywords related to the design of microservices and (2) applied forward and backward snowballing [13, 14] to minimize the risk of missing important papers.”
MethodSocial cryptocurrencies as model for enhancing sustainable development
Mora, H., Morales-Morales, M. R., Pujol-López, F. A., & Mollá-Sirvent, R. (2021) · Kybernetes · doi:10.1108/k-05-2020-0259 · Cited by 35
Obaidi, M., & Klünder, J. (2021) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3463274.3463328 · Cited by 58
“This snowball principle was performed according to Wohlin [75].”
MethodRabe, M., Milz, S., & Mäder, P. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00023 · Cited by 44
“As a final step, we performed backward snowballing [113] on the references of all identified publications.”
MethodThe impact of traceability on software maintenance and evolution: A mapping study
Tian, F., Wang, T., Liang, P., Wang, C., Khan, A. A., & Babar, M. A. (2021) · Journal of Software Evolution and Process · doi:10.1002/smr.2374 · Cited by 64
Khezami, N., Gharbi, N., Neji, B., & Braïek, N. B. (2022) · Sustainability · doi:10.3390/su142315826 · Cited by 46
“It is worthwhile to mention that forward as well as backward snowballing were applied by two authors as recommended by Wohlin [18], and we found no additional papers to be added to our final list of 769 relevant articles.”
MethodThree decades of software reference architectures: A systematic mapping study
Garcés, L., Martínez‐Fernández, S., de Oliveira, L. B. R., Valle, P. H. D., Ayala, C. P., Franch, X., & Nakagawa, E. Y. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111004 · Cited by 57
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34
“The snowballing procedure followed the process proposed by [51], with the exception that forward and backward snowballings were limited to a single iteration each.”
Method12 Ways to Empower: Designing for Children’s Digital Autonomy
Wang, G., Zhao, J., Van Kleek, M., & Shadbolt, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580935 · Cited by 34
“Of the 68 papers, 47 are conference publications (all full-length proceeding papers), 21 are journal articles.”
BackgroundJigs and Fixtures in Production: A Systematic Literature Review
Fiedler, F., Ehrenstein, J., Höltgen, C., Blondrath, A., Schäper, L., Göppert, A., & Schmitt, R. H. (2023) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2023.10.006 · Cited by 35
Moraes, K. K., Ganga, G. M. D., Filho, M. G., de Santa-Eulália, L. A., & Tortorella, G. L. (2024) · Supply Chain Management An International Journal · doi:10.1108/scm-02-2024-0138 · Cited by 40
Approaches to strategic alignment of software process improvement: A systematic literature review
Vasconcellos, F. J. S., Landre, G. B., da Cunha, J. A. O. G., Oliveira, J. L. D., Ferreira, R. A., & Vincenzi, A. M. R. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.09.030 · Cited by 40
Asmussen, C. B., & Møller, C. (2020) · Enterprise Information Systems · doi:10.1080/17517575.2020.1734240 · Cited by 46
“In a broad research domain it is also unlikely that one or a few papers are central for the research leaving out methods such as snowballing (Wohlin 2014).”
BackgroundA study of the performance of general compressors on log files
Yao, K., Li, H., Shang, W., & Hassan, A. E. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09822-x · Cited by 44
“We followed a snowball literature review strategy (Wohlin 2014) to find the related work on log compression and log analysis, in order to obtain a comprehensive view of the log data that is used in prior studies.”
MethodA systematic literature review of technical debt prioritization
Alfayez, R., Alwehaibi, W., Winn, R., Venson, E., & BOEHM, B. W. (2020) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3387906.3388630 · Cited by 59
“Forward and backward snowballing were conducted on the selected set of papers, while following the guidelines in [54].”
MethodSystematic literature review – improving business processes by implementing agile
Schmitt, A., & Hörner, S. (2021) · Business Process Management Journal · doi:10.1108/bpmj-10-2019-0422 · Cited by 34
Smithies, T. D., Toth, A. J., Dunican, I. C., Caldwell, J. A., Kowal, M., & Campbell, M. J. (2021) · SLEEP · doi:10.1093/sleep/zsab008 · Cited by 43
““Backward snowballing” refers to a technique where the reference lists of previously identified reviews or journal articles within a relevant topic are searched to obtain further relevant articles [52].”
BackgroundŞiean, A.-I., & Vatavu, R.-D. (2021) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3441852.3471212 · Cited by 42
“We used the 38 papers in two snowballing [134] procedures: backward snowballing (by looking at the lists of references of the selected papers) and forward snowballing (the citations are considered).”
MethodSystematic literature review and analysis for Arabic text steganography method practically
Roslan, N. A., Udzir, N. I., Mahmod, R., & Gutub, A. A. A. (2022) · Egyptian Informatics Journal · doi:10.1016/j.eij.2022.10.003 · Cited by 34
Razami, H. H., & Ibrahim, R. (2022) · Telematics and Informatics · doi:10.1016/j.tele.2022.101874 · Cited by 37
Peeters, R., Westra, D., van Raak, A., & Ruwaard, D. (2022) · Medical Care Research and Review · doi:10.1177/10775587221118156 · Cited by 37
“Moreover, we performed an additional forward snowballing procedure (i.e., a cited references search, see Wohlin, 2014) of Provan and Kenis (2008) in Web of Science.”
MethodEthical considerations in child-robot interactions
Langer, A., Marshall, P. J., & Levy‐Tzedek, S. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105230 · Cited by 46
“Our approach to identifying relevant literature was based on a snowballing approach as described by Wohlin (2014).”
Romberg, J., & Escher, T. (2024) · Digital Government Research and Practice · doi:10.1145/3603254 · Cited by 60
“As a second strategy to identify additional relevant literature, we employed a snowballing approach as defined by Wohlin [96].”
MethodInfluentialRequirements engineering for sustainable software systems: a systematic mapping study
Bambazek, P., Groher, I., & Seyff, N. (2023) · Requirements Engineering · doi:10.1007/s00766-023-00402-1 · Cited by 54
“Snowballing means, both, to consider the reference list of a paper (backward snowballing) and all papers that refer to the paper being examined (forward snowballing) [26, 27].”
BackgroundAgile human resource management: A systematic mapping study
Moh’d, S., Gregory, P., Barroca, L., & Sharp, H. (2024) · German Journal of Human Resource Management Zeitschrift für Personalforschung · doi:10.1177/23970022231226316 · Cited by 92
“We used snowballing by looking through the reference lists of the publications that were included (Wohlin, 2014).”
Contrasting the features and functionalities of urban microclimate simulation tools
Vurro, G., & Carlucci, S. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114042 · Cited by 44
Ethics in AI through the practitioner’s view: a grounded theory literature review
Pant, A., Hoda, R., Tantithamthavorn, C. K., & Turhan, B. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10465-5 · Cited by 66
““Snow-balling refers to using the reference list of a paper or the citations to the paper to identify additional papers” (Wohlin 2014).”
A Systematic Review of Outdoor Thermal Comfort Studies for the Urban (Re)Design of City Squares
Mandič, L., Đjukić, A., Marić, J., & Mitrović, B. (2024) · Sustainability · doi:10.3390/su16124920 · Cited by 53
“A search and identification of additional studies suitable for consideration by citation was carried out using the “snowballing” method [46].”
Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review
Bach, T. A., Kristiansen, J. K., Babić, A., & Jacovi, A. (2024) · IEEE Access · doi:10.1109/access.2024.3437190 · Cited by 62
“Future studies may be improved by implementing snowballing [68] or hand searching methods [69] to explore an additional perspective of the literature.”
BackgroundMethodThe Role of Multi-Agents in Digital Twin Implementation: Short Survey
Kalyani, Y., & Collier, R. (2024) · ACM Computing Surveys · doi:10.1145/3697350 · Cited by 54
“Google Scholar was chosen to minimise bias against any specific scientific publisher, adhering to the methodology recommended in [51].”
Rashid, N., & Khan, S. U. (2018) · Journal of Software Evolution and Process · doi:10.1002/smr.1964 · Cited by 50
“Snowballing is more prominent search mechanism that uses an intelligent way to search relevant publications by means of reference, citation, and author(s) of a paper in a relevant domain.(82)”
BackgroundA systematic mapping study on text analysis techniques in software architecture
Bi, T., Liang, P., Tang, A., & Yang, C. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.07.055 · Cited by 37
“Snowballing: We followed a snowballing technique (Wohlin, 2014) to collect and scan the references of the selected studies in order to find more potential primary studies Snowballing is an iterative process (note that we only conducted backward snowballing).”
MethodA conceptual perspective on interoperability in context-aware software systems
Motta, R. C., de Oliveira, K. M., & Travassos, G. H. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.07.001 · Cited by 47
“From the 16 selected studies, backward and forward snowballing sampling [15] were performed following the guidelines described in [19].”
BackgroundMethodSustainability and universal design aspects in heritage building refurbishment
Kristl, Ž., Salaj, A. T., & Roumboutsos, A. (2019) · Facilities · doi:10.1108/f-07-2018-0081 · Cited by 33
Global software development governance: Challenges and solutions
Manjavacas, A., Vizcaíno, A., Ruíz, F., & Piattini, M. G. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2266 · Cited by 35
Framework for technical specifications of 3D concrete printers
Chung, J.-Y., Lee, G., & Kim, J. (2021) · Automation in Construction · doi:10.1016/j.autcon.2021.103732 · Cited by 36
A systematic literature review of vehicle speed assistance in intelligent transportation system
Asadi, M., Fathy, M., Mahini, H., & Rahmani, A. M. (2021) · IET Intelligent Transport Systems · doi:10.1049/itr2.12077 · Cited by 33
“Then, the produced set of results were candidates to run forward and backward snowballing as described in [11] in which references to and from those papers were analysed to check for additional literature.”
MethodThe Case for Adaptive Security Interventions
Rauf, I., Petre, M., Tun, T. T., López, T., Lunn, P., van der Linden, D., Towse, J. N., Sharp, H., Levine, M., Rashid, A., et al. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3471930 · Cited by 42
“Forward and backward snowballing approaches [180] were used to cover, as comprehensively as possible, the studies conducted in this domain in the past 10 years, i.”
MethodJelínková, M., Tetřevová, L., Vávra, J., & Munzarová, S. (2021) · Sustainability · doi:10.3390/su13179886 · Cited by 41
“” Sources of literary research were subsequently extended thanks to the application of the snowball method [93,94].”
MethodAdeyanju, S., O'Connor, A., Addoah, T., Bayala, E. R. C., Djoudi, H., Moombe, K., Reed, J., Ros-Tonen, M. A. F., Siangulube, F. S., Sikanwe, A., et al. (2021) · The International Forestry Review · doi:10.1505/146554821833992776 · Cited by 69
A Shared Cyber Threat Intelligence Solution for SMEs
van Haastrecht, M., Golpur, G., Tzismadia, G., Kab, R., Priboi, C., David, D., Răcătăian, A., Baumgartner, L., Fricker, S., Ruiz, J., et al. (2021) · Electronics · doi:10.3390/electronics10232913 · Cited by 56
Wang, Y., Mäntylä, M., Liu, Z., & Markkula, J. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111259 · Cited by 66
“That follows some guidelines of “Systematic literature studies using snowballing” from Wohlin et al. [26] but not all.”
Alonso, I. B., Sánchez-Rivero, M. V., & Pozas, B. M. (2022) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2022.131870 · Cited by 45
When machine learning meets Network Management and Orchestration in Edge-based networking paradigms
Shahraki, A., Ohlenforst, T., & Kreyß, F. (2022) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2022.103558 · Cited by 38
A survey on software coupling relations and tools
Fregnan, E., Baum, T., Palomba, F., & Bacchelli, A. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.11.008 · Cited by 42
König, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M. (2023) · doi:10.31234/osf.io/ybu3w · Cited by 33
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34
A systematic review of the mental health of women in construction: future research directions
Rotimi, F. E., Burfoot, M., Naismith, N., Mohaghegh, M., & Brauner, M. (2022) · Building Research & Information · doi:10.1080/09613218.2022.2132905 · Cited by 45
“This research further implemented a backward snowballing by using the reference lists to identify additional papers with relevant information (Bridges et al., 2020; Wohlin, 2014).”
MethodSundaram, G., & Berleant, D. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-3243-6_7 · Cited by 47
Model driven engineering for machine learning components: A systematic literature review
Naveed, H., Arora, C., Khalajzadeh, H., Grundy, J., & Haggag, O. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107423 · Cited by 56
Zaoui, M., Belfaik, Y., Sadqi, Y., Maleh, Y., & Ouazzane, K. (2024) · IEEE Access · doi:10.1109/access.2024.3403197 · Cited by 50
“Snowballing Search: To enhance the coverage of relevant studies, the manual search was supplemented with backward and forward snowballing techniques following the snowballing procedure guidelines described in [36].”
MethodVulnerability detection techniques for smart contracts: A systematic literature review
Vidal, F. R., Ivaki, N., & Laranjeiro, N. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112160 · Cited by 65
Model-driven engineering for digital twins: a systematic mapping study
Lehner, D., Zhang, J., Pfeiffer, J., Sint, S., Splettstößer, A.-K., Wimmer, M., & Wortmann, A. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01264-7 · Cited by 64
“Snowballing: In the last step, we performed forward and backward snowballing [71] on the 65 papers included after the detailed reviewing phase.”
A literature review on the effectiveness and efficiency of business modeling
Wilson, M., Wnuk, K., Silvander, J., & Gorschek, T. (2018) · DiVA (Blekinge Institute of Technology) · doi:10.5277/e-inf180111 · Cited by 72
“[21] K. Petersen and C. Wohlin, “Context in industrial software engineering research,” in 3rd International Symposium on Empirical Software Engineering and Measurement, ESEM, 2009, pp. 401–404.”
BackgroundMethodInfluentialExtraction of Configurable and Reusable Microservices from Legacy Systems
Carvalho, L. C. D. S., Garcia, A., Assunção, W. K. G., et al. (2019) · doi:10.1145/3336294.3336319 · Cited by 32
“In addition, we performed a snowballing search [31] in studies that cited the referred mapping studies.”
MethodRogers, K., Karaosmanoglu, S., Wolf, D. A., Steinicke, F., & Nacke, L. E. (2021) · Frontiers in Virtual Reality · doi:10.3389/frvir.2021.694660 · Cited by 37
“To avoid being too dependent on the speci fi cs of our search terms, and to catch papers that describe asymmetric gameplay while not using that terminology, we applied a backwards snowball approach (Wohlin, 2014) to the 21 initially identi fi ed papers.”
MethodComprehensiveness of circular economy assessments of regions: a systematic review at the macro-level
Van Bueren, B. J. A., Iyer-Raniga, U., Leenders, M. A. A. M., & Argus, K. (2021) · Environmental Research Letters · doi:10.1088/1748-9326/ac209c · Cited by 30
“Batch 3 extends the corpus via snowballing by checking citations in papers from the search results to identify additional papers [31].”
BackgroundChabanet, S., El-Haouzi, H. B., Morin, M., Gaudreault, J., & Thomas, P. (2022) · International Journal of Production Research · doi:10.1080/00207543.2022.2068086 · Cited by 42
Surgical Tool Datasets for Machine Learning Research: A Survey
Rodrigues, M., Mayo, M., & Patros, P. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01640-6 · Cited by 42
“We also conducted other complimentary searches, such as reviewing reference lists, searching through conference proceedings, and obtaining leads from prominent researchers and authors in this area (Wohlin, 2014).”
MethodBaker, D., Briant, S., Hajirasouli, A., Yiğitcanlar, T., Paz, A., Bhaskar, A., Corry, P., Whelan, K., Donehue, P., & Parsons, H. (2022) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2022.100731 · Cited by 57
Image and Video Coding Techniques for Ultra-low Latency
Žádník, J., Makitalo, M. J., Vanne, J., & Jääskeläinen, P. (2022) · ACM Computing Surveys · doi:10.1145/3512342 · Cited by 39
“A snowballing approach [153] was used in Section 4 with the help of the pybliometrics library [119] to discover possible related works within citations and references of included papers.”
MethodBuma, L. E., Vluggen, S., Zwakhalen, S. M. G., Kempen, G. I. J. M., & Metzelthin, S. (2022) · European Journal of Ageing · doi:10.1007/s10433-022-00693-3 · Cited by 43
“An additional snowball sampling was used on studies included in the final sample (Wohlin 2014).”
MethodSahahiri, R. M., Griffin, A. L., & Sun, Q. C. (2023) · Sustainability · doi:10.3390/su15032678 · Cited by 43
Machine learning enhancing metaheuristics: a systematic review
Oliveira, A. L. D. C., Britto, A., & de Gusmão, R. P. (2023) · Soft Computing · doi:10.1007/s00500-023-08886-3 · Cited by 38
Offshore wind H&S: A review and analysis
Rowell, D., McMillan, D., & Carroll, J. (2023) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2023.113928 · Cited by 38
Games to support teaching clinical reasoning in health professions education: a scoping review
Koelewijn, G. R., Hennus, M. P., Kort, H. S. M., Frenkel, J., & van Houwelingen, T. (2024) · Medical Education Online · doi:10.1080/10872981.2024.2316971 · Cited by 51
“For the third stage, in addition to Khalil et al.’s (2016) approach, we employed forward and backward snow-balling to identify additional studies [32].”
MethodHaindl, P., Kochberger, P., & Sveggen, M. (2024) · IEEE Access · doi:10.1109/access.2024.3406500 · Cited by 45
“Third, we employed a backward snowballing step, adhering to the guidelines of Wohlin [33], to identify as many related papers as possible by manually searching the references of selected papers.”
MethodInfluentialChew, A. W. Z., He, R., & Zhang, L. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10145-z · Cited by 46
“To achieve an extensive and snowballing literature review process [37, 38] for evaluating the adoption of PIML techniques for infrastructure engineering, the following key questions must be addressed, namely: (1) What is the current state-of-the-art for PIML, in terms of its concept,…”
Belabyad, M., Kontovas, C. A., Pyne, R., & Chang, C. (2025) · Maritime Policy & Management · doi:10.1080/03088839.2025.2475177 · Cited by 62
“Additionally, we have employed the so-called, snowballing, often called as a ‘chain-referral’ method (Wohlin 2014) which involved going through the reference lists of identified articles, pinpointing and assessing other potential sources that may be pertinent to the research query (Greenhalgh and…”
Carvalho, T., Farber, S., Manaugh, K., & El-Geneidy, A. (2025) · Transport Reviews · doi:10.1080/01441647.2025.2513530 · Cited by 33
“To ensure a comprehensive coverage of relevant literature, backward snowballing was employed (Wohlin, 2014).”
Designing Interactions with Generative AI for Art and Creativity: A Systematic Review and Taxonomy
Hu, X., Xing, Y., Cai, X., Zhao, Y., Cook, M., Borgo, R., & Neate, T. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735843 · Cited by 61
“Finally, we conducted a snowballing step and another round of screening and eligibility, following the guidance of Wohlin [177].”
Wen, R., Zeng, D., Yang, Z., Jiang, L., Ma, M., Chen, B., & van Beek, T. A. (2018) · Journal of Agricultural and Food Chemistry · doi:10.1021/acs.jafc.8b02388 · Cited by 31
Commenting source code: is it worth it for small programming tasks?
Nielebock, S., Krolikowski, D., Krüger, J., Leich, T., & Ortmeier, F. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9664-z · Cited by 41
“This way, we identified an initial set of three studies: Afterwards, we applied backwards and forwards snowballing (Jalali and Wohlin 2012; Wohlin 2014) on these studies using Google Scholar to extend the scope of our review (last checked July 18 th 2018).”
MethodThe State of Empirical Evaluation in Static Feature Location
Razzaq, A., Wasala, A., Exton, C., & Buckley, J. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3280988 · Cited by 37
“To further increase the scope of the paper one iteration of snowballing was performed using the referenced articles in the 153 papers, as per the guidelines presented by Wohlin [56].”
MethodMonteiro-Guerra, F., Signorelli, G. R., Tadas, S., Zubiete, E. D., Rivera-Romero, O., Fernández-Luque, L., & Caulfield, B. (2020) · JMIR mhealth and uhealth · doi:10.2196/17552 · Cited by 33
“RenderX were identified through a snowballing process by screening the reference list of the included ones for related papers [40].”
Karni, L., Dalal, K., Memedi, M., Kalra, D., & Klein, G. O. (2020) · Journal of Medical Internet Research · doi:10.2196/17459 · Cited by 41
“We also used the snowball technique to retrieve additional relevant articles [18].”
MethodKundu, S., Kabir, M. E., Morgan, E. A., Davey, P., & Hossain, M. (2020) · Climate · doi:10.3390/cli8090098 · Cited by 35
“A snowball search uses the reference list or citations in a paper to identify additional papers [25].”
MethodExploring the meaning of usable security – a literature review
Lennartsson, M., Kävrestad, J., & Nohlberg, M. (2021) · Information and Computer Security · doi:10.1108/ics-10-2020-0167 · Cited by 47
Kreimeier, J., & Götzelmann, T. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4040079 · Cited by 39
“Here, we identified initial relevant literature using the initial keywords ‘virtual reality’, ‘virtual environment’, ‘blind’ and ‘visually impaired’ and iteratively completed (and verified) our database by a backwards snowball-search [27] from latest work to the earliest findable publication, see the complete workflow…”
BackgroundPatterns Related to Microservice Architecture: a Multivocal Literature Review
Valdivia, J. A., Lora-Gonzalez, A., Limón, X., Cortés-Verdín, K., & Ocharán-Hernández, J. O. (2020) · Programming and Computer Software · doi:10.1134/s0361768820080253 · Cited by 42
“Simplified query associated with RQ1 , RQ2 , and RQ3 : (Implementing or Developing) microservices or microservices design patterns In addition to the automated search, manual research was incorporated using the backward snow-balling method [30].”
MethodIntegrative review of Islamic marketing
Shah, S. A., Bhutto, M. H., & Azhar, S. M. (2021) · Journal of Islamic marketing · doi:10.1108/jima-07-2020-0216 · Cited by 53
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.