Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
The Circular Economy–A new sustainability paradigm?
Geissdoerfer, M., Savaget, P., Bocken, N. M. P., & Hultink, E. J. (2017) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2016.12.048 · Cited by 9,360
“This review started with a sample of relevant papers published by highly cited journals and academics, which was then followed by a semi-structured snowballing approach (Wohlin, 2014), to capture both established and emerging conceptual trends (see Figure 5).”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
“The list of selected papers was largely completed afterwards, by using a backward and forward snowballing strategy that consists of using the reference list of the selected papers and the citations to these papers to identify additional papers [65].”
MethodGuidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669
“Wohlin [61] provides some advice to this end:”
BackgroundInfluentialAli, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
A survey on digital twin: definitions, characteristics, applications, and design implications
Barricelli, B. R., Casiraghi, E., & Fogli, D. (2019) · IEEE Access · doi:10.1109/access.2019.2953499 · Cited by 1,480
“We chose Google Scholar to avoid bias in favor of any specific scientific publisher, as suggested by [42].”
MethodEmbodied GHG emissions of buildings – The hidden challenge for effective climate change mitigation
Röck, M., Saade, M. R. M., Balouktsi, M., Rasmussen, F. N., Birgisdottir, H., Frischknecht, R., Habert, G., Lützkendorf, T., & Passer, A. (2020) · Applied Energy · doi:10.1016/j.apenergy.2019.114107 · Cited by 1,163
“The collection of published information and subsequent analysis of the documents were performed following the structured protocol for Systematic Literature Review (SLR) and the ‘snowball’ approach [45,46], a complementary strategy to assure relevant sources are not left out which consists of checking the reference lis…”
MethodBusiness model innovation for circular economy and sustainability: A review of approaches
Pieroni, M. P. P., McAloone, T. C., & Pigosso, D. C. A. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.01.036 · Cited by 1,143
Circular business models: a review
Geissdoerfer, M., Pieroni, M. P. P., Pigosso, D. C. A., & Soufani, K. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.123741 · Cited by 1,089
A systematic review of gamification in e-Health
Sardi, L., Idri, A., & Fernández-Alemán, J. L. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.05.011 · Cited by 1,015
Life Expectancy and Mortality Rates in the United States, 1959-2017
Woolf, S. H., & Schoomaker, H. (2019) · JAMA · doi:10.1001/jama.2019.16932 · Cited by 930
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
“To ensure a high coverage of ML testing related papers, we therefore also performed snowballing [74] on each of the related papers found by keyword searching.”
MethodIndustry 4.0 and Health: Internet of Things, Big Data, and Cloud Computing for Healthcare 4.0
Aceto, G., Persico, V., & Pescapé, A. (2020) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2020.100129 · Cited by 876
“following the methodology described in [22].”
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
Ouyang, F., Zheng, L., & Jiao, P. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-10925-9 · Cited by 831
“After the screening of the full articles, the snowballing approach was performed based on the guidelines to find the articles that were not extracted by using the search strings (Wohlin, 2014).”
MethodA systematic mapping study on technical debt and its management
Li, Z., Avgeriou, P., & Liang, P. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2014.12.027 · Cited by 743
“Second, we employed the “snowballing” technique (Wohlin, 2014) to include the potential studies in the references of the selected studies retrieved from the database searches.”
InfluentialOperational principles of circular economy for sustainable development: Linking theory and practice
Suárez-Eiroa, B., Fernández, E., Méndez-Martínez, G., & Soto-Oñate, D. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.12.271 · Cited by 688
Tackling the plastic problem: A review on perceptions, behaviors, and interventions
Heidbreder, L. M., Bablok, I., Drews, S., & Menzel, C. (2019) · Science of The Total Environment · doi:10.1016/j.scitotenv.2019.02.437 · Cited by 680
A systematic literature review of blockchain cyber security
Taylor, P. J., Dargahi, T., Dehghantanha, A., Parizi, R. M., & Choo, K.-K. R. (2020) · Digital Communications and Networks · doi:10.1016/j.dcan.2019.01.005 · Cited by 633
Online Learning: Adoption, Continuance, and Learning Outcome—A Review of Literature
Panigrahi, R., Srivastava, P. R., & Sharma, D. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.05.005 · Cited by 607
Barriers to vaccination in Latin America: A systematic literature review
Guzman-Holst, A., DeAntonio, R., Prado-Cohrs, D., & Juliao, P. (2020) · Vaccine · doi:10.1016/j.vaccine.2019.10.088 · Cited by 594
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Next, to mitigate the risk of omitting relevant literature from this survey, we also perform backward snowballing [130] by inspecting the references cited by the collected papers so far.”
Meeting the review family: exploring review types and associated information retrieval requirements
Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12276 · Cited by 570
Ali, O., Abdelbaki, W., Shrestha, A., Elbasi, E., Alryalat, M. A. A., & Dwivedi, Y. K. (2023) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2023.100333 · Cited by 568
Kauhanen, L., Wan Mohd Yunus, W. M. A., Lempinen, L., Peltonen, K., Gyllenberg, D., Mishina, K., Gilbert, S., Bastola, K., Brown, J. S. L., & Sourander, A. (2023) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-022-02060-0 · Cited by 544
“We also used the backward snowballing technique to search for other relevant papers [17], by looking at the reference lists of the selected papers.”
MethodMachine learning techniques and data for stock market forecasting: A literature review
Kumbure, M. M., Lohrmann, C., Luukka, P., & Porras, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116659 · Cited by 535
Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review
Qian, Y., & Lehman, J. (2018) · ACM Transactions on Computing Education · doi:10.1145/3077618 · Cited by 529
“The first step of the snowballing approach is to identify a starter set of papers regarding a topic (Wohlin 2014).”
BackgroundMethodThe application of AI technologies in STEM education: a systematic review from 2011 to 2021
Xu, W., & Ouyang, F. (2022) · International Journal of STEM Education · doi:10.1186/s40594-022-00377-5 · Cited by 495
“After the preliminary screening of articles, snowballing was conducted (Wohlin, 2014) to find the articles that were not extracted using the search strings.”
MethodA Survey of DevOps Concepts and Challenges
Leite, L., Rocha, C., Kon, F., Milojicic, D., & Meirelles, P. (2020) · ACM Computing Surveys · doi:10.1145/3359981 · Cited by 484
“We also applied a snowballing process [122] to cover important work not found with the query string “devops.””
Method23 Ways to Nudge: A Review of Technology-Mediated Nudging in Human-Computer Interaction
Caraban, A., Karapanos, E., Gonçalves, D., & Campos, P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300733 · Cited by 475
“From the 71 articles, 14 papers were added following Wohlin guidelines for snowballing in systematic reviews [97].”
BackgroundA Survey of App Store Analysis for Software Engineering
Martin, W., Sarro, F., Jia, Y., Zhang, Y., & Harman, M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2630689 · Cited by 459
“2, we also perform snowballing [244] on many of the included studies.”
Almeqdad, Q. I., Alodat, A. M., Alquraan, M. F., Mohaidat, M. A., & Al-Makhzoomy, A. K. (2023) · Cogent Education · doi:10.1080/2331186x.2023.2218191 · Cited by 444
“From those studies, ten studies were identified using the backwards and forward snowballing methods (Wohlin, 2014).”
BackgroundReading from paper compared to screens: A systematic review and meta‐analysis
Clinton, V. (2019) · Journal of Research in Reading · doi:10.1111/1467-9817.12269 · Cited by 436
Wohlin, C., Kalinowski, M., Romero Felizardo, K., & Mendes, E. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106908 · Cited by 432
“Once a start set was identified, back-ward and forward snowballing were performed to determine further articles to include, using the guidelines in Wohlin (2014).”
InfluentialSelf-citationAli, O., Ally, M., Clutterbuck, & Dwivedi, Y. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102199 · Cited by 422
Feng, Z., González, V. A., Amor, R., Lovreglio, R., & Cabrera-Guerrero, G. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.09.002 · Cited by 423
“Meanwhile, another approach called snowballing (retrieve relevant papers based on target papers’ references list or paper citing) (Wohlin, 2014, May) was also adopted with Google Scholar, which indexes most scholarly literature, as a complementary method to cover any missing papers.”
MethodCryptocurrency trading: a comprehensive survey
Fang, F., Ventre, C., Basios, M., Kanthan, L., Martinez-Rego, D., Wu, F., & Li, L. (2022) · Financial Innovation · doi:10.1186/s40854-021-00321-6 · Cited by 413
“To ensure a high coverage, we adopted the so-called snowballing [222] method on each paper found through these keywords.”
MethodTailored gamification: A review of literature
Klock, A. C. T., Gasparini, I., Pimenta, M. S., & Hamari, J. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102495 · Cited by 403
“…et al., 2015; Roosta et al., 2016; Taspinar et al., 2016; Tondello et al., 2017a; 2017b; 2016; Utomo and Santoso, 2015; Čudanov et al., 2014; Xu and Tang, 2015) have gone through the backward snowballing process, which uses the reference list of each one to identify additional work (Wohlin, 2014).”
MethodResearch and application of machine learning for additive manufacturing
Qin, J., Hu, F., Liu, Y., Witherell, P., Wang, C. C. L., Rosen, D. W., Simpson, T. W., Lu, Y., & Tang, Q. (2022) · Additive Manufacturing · doi:10.1016/j.addma.2022.102691 · Cited by 391
“A manual search process is adopted to search for additional sources related to the review topic [27,28], where the exclusion and inclusion criteria were re-applied.”
Circular economy implementation in the agricultural sector: Definition, strategies and indicators
Velasco-Muñoz, J. F., Mendoza, J. M. F., Aznar-Sánchez, J. A., & Gallego-Schmid, A. (2021) · Resources, Conservation and Recycling · doi:10.1016/j.resconrec.2021.105618 · Cited by 388
Stepin, I., Alonso, J. M., Catala, A., & Pereira-Farina, M. (2021) · IEEE Access · doi:10.1109/access.2021.3051315 · Cited by 376
“The present survey has been undertaken as a systematic literature review following the guidelines by Kitchenham and Charters [58], Kitchenham et al. [59], and Wohlin [60].”
BackgroundMethodA systematic review of Green AI
Verdecchia, R., Sallou, J., & Cruz, L. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1507 · Cited by 375
“In order to enrich the set of selected primary studies, and ensure that the primary study comprehensively represents the Green AI body of literature, the automated search results are complemented with a recursive bidirectional snowballing process (Wohlin, 2014).”
MethodA systematic literature review: Opinion mining studies from mobile app store user reviews
Genc-Nayebi, N., & Abran, A. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2016.11.027 · Cited by 358
A systematic review on Asian's farmers' adaptation practices towards climate change
Shaffril, H. A. M., Krauss, S. E., & Samsuddin, S. F. (2018) · Science of The Total Environment · doi:10.1016/j.scitotenv.2018.06.349 · Cited by 347
Langer, M., & Landers, R. N. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106878 · Cited by 338
Simulation in industry 4.0: A state-of-the-art review
de Paula Ferreira, W., Armellini, F., & De Santa-Eulalia, L. A. (2020) · Computers & Industrial Engineering · doi:10.1016/j.cie.2020.106868 · Cited by 333
Explainable Artificial Intelligence in CyberSecurity: A Survey
Capuano, N., Fenza, G., Loia, V., & Stanzione, C. (2022) · IEEE Access · doi:10.1109/access.2022.3204171 · Cited by 330
A Systematic Literature Review and Meta-Analysis on Cross Project Defect Prediction
Hosseini, S., Turhan, B., & Gunarathna, D. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2770124 · Cited by 317
“As per the guidelines provided by Wohlin, we applied backward and forward snowballing iteratively until no further papers for inclusion were detected [28].”
MethodIslam, M. R., Ahmed, M. U., Barua, S., & Begum, S. (2022) · Applied Sciences · doi:10.3390/app12031353 · Cited by 311
“During the process, 19 additional related articles were found from a complementary snowballing search [43], in simpler terms, a recursive reference search.”
BackgroundResearch on Architecting Microservices: Trends, Focus, and Potential for Industrial Adoption
Francesco, P. D., Malavolta, I., & Lago, P. (2017) · 2017 IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa.2017.24 · Cited by 311
The ai-based cyber threat landscape: A survey
Kaloudi, N., & Li, J. (2021) · ACM Computing Surveys · doi:10.1145/3372823 · Cited by 310
“Then, we complemented the manual search with backward and forward snowballing technique of the papers that belonged to the three defined categories for a better coverage of related studies, using the snowballing procedure guidelines [84].”
MethodAn approach to business model innovation and design for strategic sustainable development
França, C. L., Broman, G., Robèrt, K.-H., Basile, G., & Trygg, L. (2017) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2016.06.124 · Cited by 309
Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., & Chatzigeorgiou, A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.10.006 · Cited by 305
A systematic mapping review of surrogate safety assessment using traffic conflict techniques
Arun, A., Haque, M. M., Bhaskar, A., Washington, S., & Sayed, T. (2021) · Accident Analysis & Prevention · doi:10.1016/j.aap.2021.106016 · Cited by 305
AI in Education: A Systematic Literature Review
Tahiru, F. (2021) · Journal of Cases on Information Technology · doi:10.4018/jcit.2021010101 · Cited by 299
Business models for sustainable consumption in the circular economy: An expert study
Tunn, V. S. C., Bocken, N. M. P., van den Hende, E. A., & Schoormans, J. P. L. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.11.290 · Cited by 298
A Comparative Study: Blockchain Technology Utilization Benefits, Challenges and Functionalities
Ali, O., Jaradat, A., Kulakli, A., & Abuhalimeh, A. (2021) · IEEE Access · doi:10.1109/access.2021.3050241 · Cited by 294
“It is applied to utilize the rundown of references to recognize the new articles suitable for inclusion [54].”
MethodAzeem, M. I., Palomba, F., Shi, L., & Wang, Q. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.12.009 · Cited by 289
“c We performed a snowballing process to search for possible missing papers [67].”
BackgroundMethodInfluentialBuilding design and construction strategies for a circular economy
Eberhardt, L. C. M., Birkved, M., & Birgisdottir, H. (2022) · Architectural Engineering and Design Management · doi:10.1080/17452007.2020.1781588 · Cited by 295
Auditing Large Language Models: A Three-Layered Approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023) · doi:10.2139/ssrn.4361607 · Cited by 292
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
“Hence, we used a snowballing technique [293], i.”
MethodDeeva, G., Bogdanova, D., Serral, E., Snoeck, M., & De Weerdt, J. (2021) · Computers & Education · doi:10.1016/j.compedu.2020.104094 · Cited by 289
BIM and LCA Integration: A Systematic Literature Review
Potrč Obrecht, T., Röck, M., Hoxha, E., & Passer, A. (2020) · Sustainability · doi:10.3390/su12145534 · Cited by 286
“Additionally, we collected further relevant literature via the “snowball” approach [13], by checking the references of the collected papers and interviewing the experts in this field, as done by [14,15].”
MethodArchitecting with microservices: A systematic mapping study
Di Francesco, P., Lago, P., & Malavolta, I. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.01.001 · Cited by 277
“In this phase we complemented the automatic search with a closed recursive backward and forward snowballing activity ( Wohlin, 2014 ).”
MethodTedre, M., Toivonen, T., Kahila, J., Vartiainen, H., Valtonen, T., Jormanainen, I., & Pears, A. (2021) · IEEE Access · doi:10.1109/access.2021.3097962 · Cited by 274
“Second, in order to extend the overview of the field, snowball sampling was applied to the bibliography sections of the articles already identified [117].”
MethodThe logistics of the short food supply chain: A literature review
Paciarotti, C., & Torregiani, F. (2021) · Sustainable Production and Consumption · doi:10.1016/j.spc.2020.10.002 · Cited by 262
Jaramillo-Yánez, A., Benalcázar, M. E., & Mena-Maldonado, E. (2020) · Sensors · doi:10.3390/s20092467 · Cited by 259
“, backward snowballing) [54] (see Table 2).”
MethodHuman Activity Recognition Using Inertial Sensors in a Smartphone: An Overview
Sousa Lima, W., Souto, E., El-Khatib, K., Jalali, R., & Gama, J. (2019) · Sensors · doi:10.3390/s19143213 · Cited by 256
“In addition to the automated search in the digital libraries, we also used the snowballing technique [35] give us more security in capturing relevant articles.”
MethodTesting machine learning based systems: a systematic mapping
Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09881-0 · Cited by 251
“Since database search and study selection could miss relevant studies, in the citation-based search step the assessors complemented the obtained pool of studies with those reached via snowballing [131] (i.”
BackgroundAI-Based Intrusion Detection Systems for In-Vehicle Networks: A Survey
Rajapaksha, S., Kalutarage, H., Al-Kadri, M. O., Petrovski, A., Madzudzo, G., & Cheah, M. (2023) · ACM Computing Surveys · doi:10.1145/3570954 · Cited by 244
“• Backward and forward snowballing [172] and recommendations given by Mendeley reference manager were also used to collect all the relevant references.”
BackgroundMethodHe, J., Treude, C., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712003 · Cited by 246
“We repeated the snowballing process until reaching a transitive closure fixed point, where no new relevant papers were found, resulting in an additional 30 papers identified.”
LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
He, J., Treude, C., & Lo, D. (2024) · arXiv · doi:10.48550/arxiv.2404.04834 · Cited by 246
Garousi, V., Petersen, K., & Ozkan, B. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.07.006 · Cited by 241
“To decrease the risk of missing relevant studies, similar to previous SM and SLR studies, our search strategy also included forward and backward snowballing using the guidelines from [16, 17].”
MethodLwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241
Cloud computing-enabled healthcare opportunities, issues, and applications: A systematic review
Ali, O., Shrestha, A., Soar, J., & Wamba, S. F. (2018) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2018.07.009 · Cited by 232
Critical success factors for modular integrated construction projects: a review
Wuni, I. Y., & Shen, G. Q. (2020) · Building Research & Information · doi:10.1080/09613218.2019.1669009 · Cited by 237
Tannins extraction: A key point for their valorization and cleaner production
de Hoyos-Martínez, P. L., Merle, J., Labidi, J., & Charrier – El Bouhtoury, F. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2018.09.243 · Cited by 234
“To this sample of selected and categorized articles, a semi structured snowballing approach followed (Wohlin and Claes, 2014).”
MethodChallenges and solutions when adopting DevSecOps: A systematic review
Rajapakse, R. N., Zahedi, M., Babar, M. A., & Shen, H. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2021.106700 · Cited by 235
“We then performed backward, forward snowballing [44] and snowballed the primary study lists of other secondary studies (Table 1) to capture the missed papers.”
MethodArtificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“This process is iteratively repeated until no new relevant scholarly documents are found [30, 31].”
Keding, C. (2021) · Management Review Quarterly · doi:10.1007/s11301-020-00181-x · Cited by 231
“Given the dynamics of the research field, I constantly tracked the relevant references of the articles that fulfilled our selection criteria in line with the guidelines for snowballing in systematic literature studies (Wohlin 2014) to add further concepts of relevance during the writing process.”
MethodA systematic review of AI role in the educational system based on a proposed conceptual framework
Xu, W., & Ouyang, F. (2022) · Education and Information Technologies · doi:10.1007/s10639-021-10774-y · Cited by 230
“After completing the screening of the full articles, snow-balling was performed based on the guidelines described by Wohlin (2014) to find the articles that were not extracted using the search strings.”
MethodImprovements in the StArt tool to better support the systematic review process
Fabbri, S., Silva, C., Hernandes, E., Octaviano, F., Di Thommazo, A., & Belgamo, A. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916013 · Cited by 226
“For running the snowbaling backward technique [10], StArt gets references using the ful text of the primary study.”
BackgroundMethodInfluentialNeumann, T. (2021) · Management Review Quarterly · doi:10.1007/s11301-020-00193-7 · Cited by 225
“…journals of particular relevance for the discussion were manually searched.3 Second, meta-studies and literature reviews on related topics were screened for additional studies.4 And finally, based on the guidelines of Wohlin (2014), an iterative back- and forward snowballing approach was conducted.”
Methodde Oliveira, C. T., Dantas, T. E. T., & Soares, S. R. (2021) · Sustainable Production and Consumption · doi:10.1016/j.spc.2020.11.024 · Cited by 226
Experiences from using snowballing and database searches in systematic literature studies
Badampudi, D., Wohlin, C., & Petersen, K. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745818 · Cited by 221
“More recently, in 2014, Wohlin conducted a replication study [12] to compare SB results including BSB and FSB and DB search.”
BackgroundMethodInfluentialSelf-citationDeep Learning for Phishing Detection: Taxonomy, Current Challenges and Future Directions
Do, N. Q., Selamat, A., Krejcar, O., Herrera-Viedma, E., & Fujita, H. (2022) · IEEE Access · doi:10.1109/access.2022.3151903 · Cited by 221
“This SLR applied a paper selection process based on PRISMA guidelines [25] which consists of several stages, such as automatic search, duplicity removal, title and abstract screening, full-text selection, and snowballing [26].”
MethodFavoretto, C., Mendes, G. H. S., Oliveira, M. G., Cauchick-Miguel, P. A., & Coreynen, W. (2022) · Industrial Marketing Management · doi:10.1016/j.indmarman.2022.01.003 · Cited by 219
“Finally, to overcome the potential limitations of the search string, a backward snowball process based on Wohlin (2014) was performed, which resulted in the selection of five additional articles, mainly articles already included in other DS reviews (e.g., Allmendinger & Lombreglia, 2005; Eloranta &…”
MethodSalvo, D., Garcia, L., Reis, R. S., Stankov, I., Goel, R., Schipperijn, J., Hallal, P. C., Ding, D., & Pratt, M. (2021) · Journal of Physical Activity and Health · doi:10.1123/jpah.2021-0413 · Cited by 218
“These strategies can help improve both physical activity and achievement of SDGs.”
MethodInfluentialThe quest for a circular economy final definition: A scientific perspective
Nobre, G. C., & Tavares, E. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.127973 · Cited by 210
Reviewing Paradox Theory in Corporate Sustainability Toward a Systems Perspective
Carmine, S., & De Marchi, V. (2023) · Journal of Business Ethics · doi:10.1007/s10551-022-05112-2 · Cited by 212
“The snowballing procedure described by Wohlin (2014) has two main phases.”
BackgroundMethodInfluentialState of the art of cyber-physical systems security: An automatic control perspective
Zacchia Lun, Y., D’Innocenzo, A., Smarra, F., Malavolta, I., & Di Benedetto, M. D. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2018.12.006 · Cited by 211
“Maximum likelihood Bi and Zhang (2013, 2014); Bobba et al. (2010); Choi and Xie (2013); Deka et al.”
BackgroundMethodInfluentialFeliciano, D. (2019) · Sustainable Development · doi:10.1002/sd.1923 · Cited by 209
“It refers to using the reference list of an article or the citations to the article to identify additional articles (Wohlin, 2014).”
BackgroundAlbertsen, L., Richter, J. L., Peck, P., Dalhammar, C., & Plepys, A. (2021) · Resources, Conservation and Recycling · doi:10.1016/j.resconrec.2021.105658 · Cited by 210
Human resources management 4.0: Literature review and trends
da Silva, L. B. P., Soltovski, R., Pontes, J., Treinta, F. T., Leitão, P., Mosconi, E., de Resende, L. M. M., & Yoshino, R. T. (2022) · Computers & Industrial Engineering · doi:10.1016/j.cie.2022.108111 · Cited by 211
Serverless Computing: State-of-the-Art, Challenges and Opportunities
Li, Y., Lin, Y., Wang, Y., Ye, K., & Xu, C. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2022.3166553 · Cited by 211
“Finally, we adopted the often-used snowballing method [142] in software engineering to facilitate the searching process.”
MethodOn the use of deep learning in software defect prediction
Giray, G., Bennin, K. E., Köksal, Ö., Babur, Ö., & Tekinerdogan, B. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111537 · Cited by 209
A systematic review of natural language processing applied to radiology reports
Casey, A., Davidson, E., Poon, M., Dong, H., Duma, D., Grivas, A., Grover, C., Suárez-Paniagua, V., Tobin, R., Whiteley, W., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01533-7 · Cited by 206
Architectural Patterns for Microservices: A Systematic Mapping Study
Taibi, D., Lenarduzzi, V., & Pahl, C. (2018) · Proceedings of the 8th International Conference on Cloud Computing and Services Science · doi:10.5220/0006798302210232 · Cited by 207
“Paper extracted from the bibliographic sources 2754 10 random papers independently classified by three researchers Sift based on title and abstract 2669 Good inter-rater agreement on first sift (Kstatistic test) Primary papers identified 85 Secondary papers inclusion 858 855 Systematic snowballing (Wohlin, 2014) inclu…”
BackgroundMethodInfluentialMachine learning applied to software testing: A systematic mapping study
Durelli, V. H. S., Durelli, R. S., Borges, S. S., Endo, A. T., Eler, M. M., Dias, D. R. C., & Guimaraes, M. P. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2019.2892517 · Cited by 207
“Throughout the backward snowballing process, we followed the guidelines provided by Wohlin [35].”
BackgroundMethodA Survey on Unmanned Surface Vehicles for Disaster Robotics: Main Challenges and Directions
Jorge, V. A. M., Granada, R., Maidana, R. G., Jurak, D. A., Heck, G., Negreiros, A. P. F., dos Santos, D. H., Gonçalves, L. M. G., & Amory, A. M. (2019) · Sensors · doi:10.3390/s19030702 · Cited by 202
“Then, we performed a snowball search approach [52] to cover references and citing work.”
MethodEmodi, N. V., Lovell, H., Levitt, C., & Franklin, E. (2021) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2021.06.212 · Cited by 203
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.