Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 301–400 of 3,934
Challenges and Common Solutions in Smart Contract Development
Kannengießer, N., Lins, S., Sander, C., Winter, K., Frey, H., & Sunyaev, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3116808 · Cited by 122
“First, we conducted extensive literature reviews and expert interviews following established methodological guidelines [30], [58], [59] to identify challenges and solutions in smart contract development.”
BackgroundMethodInfluentialAd avoidance in the digital context: A systematic literature review and research agenda
Çelik, F., Çam, M. S., & Köseoğlu, M. A. (2022) · International Journal of Consumer Studies · doi:10.1111/ijcs.12882 · Cited by 132
Market Approaches to the Multi-Robot Task Allocation Problem: a Survey
Quinton, F., Grand, C., & Lesire, C. (2023) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-022-01803-0 · Cited by 108
“i=1 SAi (1) In addition to the selection based on the screening and qualitative assessment processes, we also performed a snowballing process, that is, we analyzed the bibliography of the selected papers to search for additional relevant papers [29].”
MethodPerkusich, M., Silva, L. C. E., Costa, A., Ramos, F., Saraiva, R., Freire, A., Dilorenzo, E., Dantas, E., Santos, D. F. S., Gorgônio, K., et al. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106241 · Cited by 132
Gender budgeting in public financial management: a literature review and research agenda
Polzer, T., Nolte, I. M., & Seiwald, J. (2021) · International Review of Administrative Sciences · doi:10.1177/00208523211031796 · Cited by 103
“Sharp and Broomhill, 1990), we conducted a snowball search of the lists of references of identified articles (Wohlin, 2014).”
MethodFrom Monolith to Microservices: A Classification of Refactoring Approaches
Fritzsch, J., Bogner, J., Zimmermann, A., & Wagner, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-06019-0_10 · Cited by 218
“Guidelines recommend to use a snowballing activity applied on the list resulting from the initial selection [40].”
Immersive Analytics with Abstract 3D Visualizations: A Survey
Kraus, M., Fuchs, J., Sommer, B., Klein, K., Engelke, U., Keim, D. A., & Schreiber, F. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14430 · Cited by 92
“To broaden the scope of our initial semi-automatic sampling strategy, we used a snowball sampling technique [Woh14].”
MethodSoftware architecture for quantum computing systems — A systematic review
Khan, A. A., Ahmad, A., Waseem, M., Liang, P., Fahmideh, M., Mikkonen, T., & Abrahamsson, P. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111682 · Cited by 133
A survey on blockchain technology in the maritime industry: Challenges and future perspectives
Farah, M. A. B., Ahmed, Y. H., Mahmoud, H., Shah, S. A., Al-Kadri, M. O., Taramonli, S., Bellekens, X., Abozariba, R., Idrissi, M., & Aneiba, A. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.03.046 · Cited by 139
A Survey of Performance Optimization for Mobile Applications
Hort, M., Kechagia, M., Sarro, F., & Harman, M. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3071193 · Cited by 91
“After a collection of publications is obtained from the repository search, we proceed to inspect the related work of the publications selected in the previous search to gather cited publications using snowballing [44].”
MethodAgile trends in Chinese global software development industry: Fuzzy AHP based conceptual mapping
Khan, A. A., Shameem, M., Nadeem, M., & Akbar, M. A. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107090 · Cited by 102
Giuffrida, M., Jiang, H., & Mangiaracina, R. (2021) · International Journal of Logistics Management · doi:10.1108/ijlm-04-2020-0158 · Cited by 68
A systematic literature review of game-based learning in Artificial Intelligence education
Zhan, Z., Tong, Y., Lan, X., & Zhong, B. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2022.2115077 · Cited by 90
“Snowballing method (Wohlin, 2014) refers to the use of references or citations of a paper to identify additional papers and add relevant information from them, where backward snowballing means using the reference list to identify additional papers to be included in the dataset and forward…”
MethodMaqbool, B., & Herold, S. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111881 · Cited by 158
Arrizubieta, J. I., Arrien, O. U., Ostolaza, M., & Múgica, A. (2020) · Metals · doi:10.3390/met10020261 · Cited by 110
“In addition, the snowball approach was used, which allows a wider range of searches based on the reference list of a paper or the citations to identify relevant publications [12,13].”
MethodData Anonymization for Pervasive Health Care: Systematic Literature Mapping Study
Zuo, Z., Watson, M., Budgen, D., Hall, R., Kennelly, C., & Moubayed, N. A. (2021) · JMIR Medical Informatics · doi:10.2196/29871 · Cited by 125
“In addition, to maximize search coverage, we conducted forward and backward snowball sampling [88] (snowball sampling refers to using the reference list of a selected paper [backward snowballing] or the citations of a selected paper [forward snowballing]) on the selected studies.”
Transfer learning in agriculture: a review
Hossen, M. I., Awrangjeb, M., Pan, S., & Mamun, A. A. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11081-x · Cited by 157
“● Snowballing search: To address the possibility of relevant studies being missed by automated search, we incorporated backward and forward snowballing (Wohlin 2014) techniques, which involve reviewing a paper’s references (backward snowballing) and the papers that cite it (forward snowballing).”
Aljedaani, W., Krasniqi, R., Aljedaani, S., Mkaouer, M. W., Ludi, S. A., & Al-Raddah, K. (2022) · Universal Access in the Information Society · doi:10.1007/s10209-022-00897-5 · Cited by 126
“If sufficient domain expertise is available, the search queries can be created manually; otherwise, a snowballing technique can be used [49, 100] in which a small number of initial search terms are used to retrieve a set of results, and then, commonly occurring domain-specific phrases are identified and used to seed f…”
BackgroundMethodMachine Learning and Data Analytic Techniques in Digital Water Metering: A Review
Rahim, M. S., Nguyen, K. A., Stewart, R. A., Giurco, D., & Blumenstein, M. (2020) · Water · doi:10.3390/w12010294 · Cited by 97
“Furthermore, some articles were selected using the ‘snowballing’ technique [30]; thus, the total number of articles expanded to 105.”
BackgroundMethodDockless electric scooters: A review of a growing micromobility mode
Orozco-Fontalvo, M., Llerena, L., & Cantillo, V. (2022) · International Journal of Sustainable Transportation · doi:10.1080/15568318.2022.2044097 · Cited by 87
Architectural Tactics for Big Data Cybersecurity Analytics Systems: A Review
Ullah, F., & Babar, M. A. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.01.051 · Cited by 113
Capturing violence in the night-time economy: A review of established and emerging methodologies
Philpot, R., Liebst, L. S., Møller, K., Lindegaard, M. R., & Levine, M. (2019) · Aggression and Violent Behavior · doi:10.1016/j.avb.2019.02.004 · Cited by 100
“Given the interdisciplinary and extensive nature of the NTE violence field, we used Google Scholar, which is known to provide the most comprehensive coverage of the research literature with the least publisher bias (MartínMartín, Orduna-Malea & López-Cózar, 2018; Wohlin, 2014).”
MethodThe impact of climate variability and change on the energy system: A systematic scoping review
Emodi, N. V., Chaiechi, T., & Beg, A. B. M. R. A. (2019) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2019.04.294 · Cited by 93
Life cycle assessment of roads: Exploring research trends and harmonization challenges
Hoxha, E., Vignisdóttir, H. R., Barbieri, D. M., Wang, F., Bohne, R. A., Kristensen, T. S., & Passer, A. (2020) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2020.143506 · Cited by 104
Trautwein, C. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.125330 · Cited by 120
Mobile cloud computing models security issues: A systematic review
Al‐Ahmad, A., Kahtan, H., Alzoubi, Y. I., Ali, O., & Jaradat, A. (2021) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2021.103152 · Cited by 100
3D Printing Technologies in Architectural Design and Construction: A Systematic Literature Review
Žujović, M., Obradović, R., Rakonjac, I., & Milošević, J. (2022) · Buildings · doi:10.3390/buildings12091319 · Cited by 112
“According to guidelines for snowballing in systematic literature studies [36], using references and citations of the analyzed papers for finding new literature is called backward and forward snowballing.”
BackgroundMethodPedestrian Behavior Prediction Using Deep Learning Methods for Urban Scenarios: A Review
Zhang, C., & Berger, C. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3281393 · Cited by 115
Early software defect prediction: A systematic map and review
Özakıncı, R., & Tarhan, A. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.06.025 · Cited by 90
Quatrini, S. (2021) · Ecosystem Services · doi:10.1016/j.ecoser.2020.101240 · Cited by 102
“To this end, different types of knowledge resources were identified using a snowballing technique (Wohlin, 2014) and screened against exclusion criteria.”
MethodCarlsson‐Kanyama, A., Hedin, B., & Katzeff, C. (2021) · Sustainability · doi:10.3390/su132212599 · Cited by 106
“After removing duplicates and applying the inclusion and exclusion criteria, we then used backward snowballing to search the references in the included papers for other relevant titles, and then forward snowballing to search for possible relevant papers referencing the included papers, as per the guidelines supplied b…”
MethodSwarm Intelligence inspired Intrusion Detection Systems — A systematic literature review
Nasir, M. H., Khan, S. A., Khan, M. M., & Fatima, M. (2022) · Computer Networks · doi:10.1016/j.comnet.2021.108708 · Cited by 99
A systematic review on search based mutation testing
Silva, R. A., Souza, S. R. S., & de Souza, P. S. L. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.01.017 · Cited by 90
A Systematic Review of Data Quality in CPS and IoT for Industry 4.0
Goknil, A., Nguyen, P., Sen, S., Politaki, D., Niavis, H., Pedersen, K. J., Suyuthi, A., Anand, A., & Ziegenbein, A. (2023) · ACM Computing Surveys · doi:10.1145/3593043 · Cited by 84
Finding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
“Jalali and Wohlin (Jalali and Wohlin 2012; Wohlin 2014) applied backward snowballing to search for primary studies in SE SLRs and found comparably good result as database search.”
ResultInfluentialAl-Ghareeb, A. Z., Cooper, S. J., & McKenna, L. (2017) · Clinical Simulation in Nursing · doi:10.1016/j.ecns.2017.05.015 · Cited by 110
Reumerman, M. O., Tichelaar, J., Piersma, B., Richir, M. C., & van Agtmael, M. A. (2018) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-018-2500-y · Cited by 109
Automatic software refactoring: a systematic literature review
Baqais, A. A. B., & Alshayeb, M. (2019) · Software Quality Journal · doi:10.1007/s11219-019-09477-y · Cited by 94
“We further performed the backward snowballing technique (Wohlin 2014) by searching the references of the selected papers.”
MethodAn empirical analysis of the costs of clone- and platform-oriented software reuse
Krüger, J., & Berger, T. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409684 · Cited by 101
“All 38 publications we identified through the first three sources served as a starting set for a snowballing search [107], and we validated our results against related work (fifth source).”
ResultA survey on healthcare data security in wireless body area networks
Jabeen, T., Ashraf, H., & Ullah, A. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-020-02728-y · Cited by 93
“A systematic literature review is performed in the study (Kitchenham 2008; Okoli 2010; Wohlin 2014) which are followed to support the proposed SLR in survey.”
MethodEffects of martial arts and combat sports training on anger and aggression: A systematic review
Lafuente, J. C., González, M. Z., & García, C. G. (2021) · Aggression and Violent Behavior · doi:10.1016/j.avb.2021.101611 · Cited by 144
“The backward snowballing technique implied looking in the reference sections of the studies potentially eligible for inclusion, while the forward snowballing technique implied exploring those studies citing the studies potentially eligible for inclusion (Wohlin, 2014).”
MethodSensing Apps and Public Data Sets for Digital Phenotyping of Mental Health: Systematic Review
Mendes, J. P. M., de Moura, I. R., van de Ven, P. W. J., Viana, D., Silva, F., Coutinho, L., Teixeira, S. S., Rodrigues, J. J. P. C., & Teles, A. S. (2021) · Journal of Medical Internet Research · doi:10.2196/28735 · Cited by 108
“This process consisted of 3 sequential phases: (1) study screening by means of metadata analysis (ie, title, abstract, and keywords); (2) full-text analysis of the articles selected in the screening phase; and (3) conducting backward snowballing [28].”
MethodMatanó, A., de Ruiter, M., Koehler, J., Ward, P. J., & Loon, A. F. V. (2022) · Earth s Future · doi:10.1029/2022ef002747 · Cited by 107
“For the delineation of a preliminary event timeline, a backward snowballing method was applied (Wohlin, 2014).”
MethodTesting embedded software: A survey of the literature
Garousi, V., Felderer, M., Karapıçak, Ç. M., & Yılmaz, U. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.06.016 · Cited by 111
Strategic Aspects of Asset Management: An Overview of Current Research
Gavrikova, E., Volkova, I., & Burda, Y. D. (2020) · Sustainability · doi:10.3390/su12155955 · Cited by 156
“We applied the snowballing technique [17] to identify keywords that pertain to the broad field of research on asset management and further used them to select articles for analysis.”
MethodEffect of chatbot-assisted language learning: A meta-analysis
Zhang, S., Shan, C., Lee, J. S. Y., Che, S., & Kim, J. H. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11805-6 · Cited by 80
“The snowballing strategy (Wohlin, 2014) was implemented after compiling all pertinent studies to locate additional appropriate studies based on reference lists in the papers from the electronic search.”
MethodChallenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review
Mehmood, E., & Anees, T. (2020) · IEEE Access · doi:10.1109/access.2020.3005268 · Cited by 127
“After performing quality assessment, we conducted back-ward snowballing [13] through reference list of each final-ized study to extract papers.”
MethodTowards a value stream perspective of circular business models
Galvão, G. D. A., Homrich, A. S., Geissdoerfer, M., Evans, S., Ferrer, P. S. S., & de Carvalho, M. M. (2020) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2020.105060 · Cited by 111
Security and Privacy Requirements for the Internet of Things: A Survey
Alhirabi, N., RANA, O. F., & Perera, C. (2021) · ACM Transactions on Internet of Things · doi:10.1145/3437537 · Cited by 96
“…requirements +security +privacy) Scopus: TITLE-ABS-KEY-AUTH(“non functional” AND requirements AND IoT) AND (LIMIT-TO (SUBJAREA,“COMP”) OR LIMIT-TO (SUBJAREA,“ENGI”)) AND (LIMIT-TO (LANGUAGE,“English”)) Scholar was used as a starting point to avoid bias in favour of any specific publisher [122].”
MethodFarrow, R. (2023) · Learning, Media and Technology · doi:10.1080/17439884.2023.2185630 · Cited by 81
“A purposive, emergent snowballing approach (Wohlin 2014; Lecy and Beatty 2012) was used to compile resources, supplemented by keyword searches on Google Scholar (n.d.) and question queries submitted to the Elicit (n.d.) database.”
FAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80
“Snowballing specifically, is a widely studied technique [11, 15, 39], which takes advantage of the relevance connection between papers and their references, could find relevant papers missed by keyword search.”
BackgroundSYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing
van Haastrecht, M., Sarhan, I., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.685591 · Cited by 80
“Additional relevant research can be identi fi ed from the reference lists (backward snowballing) and citations (forward snowballing) of included papers (Wohlin, 2014).”
BackgroundMethodInfluentialNLP-assisted software testing: A systematic mapping of the literature
Garousi, V., Bauer, S., & Felderer, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106321 · Cited by 101
“1 Selecting the source engines and search process We selected the source engines and conducted the search process using the established process for performing SLR studies in software engineering, and the established guidelines [22-25].”
BackgroundMethodInfluentialSurvey of Smart Parking Systems
Ogás, M. G. D., Fabregat, R., & Aciar, S. V. (2020) · Applied Sciences · doi:10.3390/app10113872 · Cited by 107
“After obtaining a series of primary sources, these were then processed using the snowballing search technique following the guidelines proposed by Wohlin [12].”
MethodA Review of Peer Code Review in Higher Education
Indriasari, T. D., Luxton-Reilly, A., & Denny, P. (2020) · ACM Transactions on Computing Education · doi:10.1145/3403935 · Cited by 80
“Snowballing refers to the identification of additional papers using the reference list of an article or the citations to the article [83].”
BackgroundPolicy evaluation and efficiency: a systematic literature review
Mergoni, A., & Witte, K. D. (2021) · International Transactions in Operational Research · doi:10.1111/itor.13012 · Cited by 124
“(2020), Xin-gang et al. (2020), Johnes and Virmani (2020), Titl and De Witte (2021) 9 Compare CRS-VRS Glass et al.”
BackgroundA feature-based survey of model view approaches
Brunelière, H., Burger, E., Cabot, J., et al. (2017) · Software & Systems Modeling · doi:10.1007/s10270-017-0622-9 · Cited by 80
“ward snowballing [68] by investigating the references they do cite as well as using Google Scholar9 to search for papers citing the already found papers, respectively.”
MethodMulti-period reverse logistics network design for used refrigerators
John, S. T., Sridharan, R., Kumar, P. N. R., & Krishnamoorthy, M. (2017) · Applied Mathematical Modelling · doi:10.1016/j.apm.2017.09.053 · Cited by 86
“We adopted a backward and forward snowballing approach, starting with seminal works, in order to search representative papers that explain trends and gaps in the field (Wohlin [69]).”
MethodSustainability requirements for eLearning systems: a systematic literature review and analysis
Alharthi, A. D., Spichkova, M., & Hamilton, M. (2018) · Requirements Engineering · doi:10.1007/s00766-018-0299-9 · Cited by 97
“We followed a backward iteration of snowballing procedure in [162] and refined them based on step 2 and step 3 criteria.”
MethodA survey of self-admitted technical debt
Sierra, G., Shihab, E., & Kamei, Y. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.02.056 · Cited by 97
“2 Once we identified a paper related to SATD, we applied a snowall approach to find other relevant cited work ( Wohlin, 2014 ).”
MethodRevaluing the costs and benefits of energy efficiency: A systematic review
Kamal, A., Al‐Ghamdi, S. G., & Koç, M. (2019) · Energy Research & Social Science · doi:10.1016/j.erss.2019.03.012 · Cited by 93
Vassie, C., Smith, S., & Leedham-Green, K. E. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-033480 · Cited by 87
Gamification of student peer review in education: A systematic literature review
Indriasari, T. D., Luxton-Reilly, A., & Denny, P. C. (2020) · Education and Information Technologies · doi:10.1007/s10639-020-10228-x · Cited by 103
“Snowballing refers to the identification of additional papers using the reference list of an article or the citations to the article (Wohlin 2014).”
MethodBobermin, M. P., Silva, M. M., & Ferreira, S. (2020) · Accident Analysis & Prevention · doi:10.1016/j.aap.2020.105923 · Cited by 82
Challenges of a Healthy Built Environment: Air Pollution in Construction Industry
Wieser, A. A., Scherz, M., Passer, A., & Kreiner, H. (2021) · Sustainability · doi:10.3390/su131810469 · Cited by 108
“, by reviewing the references of the collected articles [29].”
BackgroundAndroid Source Code Vulnerability Detection: A Systematic Literature Review
Senanayake, J., Kalutarage, H., Al-Kadri, M. O., Petrovski, A., & Piras, L. (2023) · ACM Computing Surveys · doi:10.1145/3556974 · Cited by 126
“The snowballing process [159] was also performed, considering all the references presented in the retrieved papers and evaluating all the papers referencing the retrieved ones, which resulted in 2 additional relevant papers.”
MethodA first systematic mapping study on combinatorial interaction testing for software product lines
Lopez-Herrejon, R. E., Fischer, S., Ramler, R., & Egyed, A. F. (2015) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2015.7107435 · Cited by 85
Carbon nanomaterials as potential substitutes for scarce metals
Arvidsson, R., & Sandén, B. A. (2017) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2017.04.048 · Cited by 77
Sobrinho, Á., Queiroz, A. C. M. D. S., da Silva, L. D., Costa, E. D. B., Pinheiro, M. E., & Perkusich, Â. (2020) · IEEE Access · doi:10.1109/access.2020.2971208 · Cited by 92
“The protocol was structured based on the SLR guidelines in software engineering proposed byKitchenham and Charters [24] and Wohlin [25].”
MethodThe evolution of minimally invasive tissue sampling in postmortem examination: a narrative review
Paganelli, C. R., Goco, N. J., McClure, E. M., Banke, K., Blau, D. M., Breiman, R. F., Menéndez, C., Rakislova, N., & Bassat, Q. (2020) · Global Health Action · doi:10.1080/16549716.2020.1792682 · Cited by 81
“A snowball methodology was applied to identify publications for this review [16,17].”
MethodEvaluating equity and justice in low-carbon energy transitions
Kime, S., Jacome, V., Pellow, D. N., & Deshmukh, R. (2023) · Environmental Research Letters · doi:10.1088/1748-9326/ad08f8 · Cited by 97
“The snowball citation approach was used to identify additional potentially relevant papers from reference lists and citations to expand the study (Wohlin 2014).”
MethodChatGPT: A brief narrative review
Gupta, B., Mufti, T., Sohail, S. S., & Madsen, D. Ø. (2023) · Cogent Business & Management · doi:10.1080/23311975.2023.2275851 · Cited by 108
“In conducting this narrative literature review, we employed a snowballing type of methodology that combined both backward and forward snowballing techniques (Felizardo et al., 2016; Wohlin, 2014).”
MethodChallenges in detecting security threats in WoT: a systematic literature review
Sardar, R., Anees, T., Al‐Shamayleh, A. S., Mehmood, E., Khalil, W., Akhunzada, A., & Shaikh, F. S. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11176-z · Cited by 80
Liu, X., & Zhong, B. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100741 · Cited by 78
Investigating the Use of a Hybrid Search Strategy for Systematic Reviews
Mourao, E., Kalinowski, M., Murta, L., Mendes, E., & Wohlin, C. (2017) · 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1109/esem.2017.30 · Cited by 77
“The iterations suggested in [16] are not included in Fig.”
BackgroundMethodInfluentialSelf-citationAgile Methodologies in Education: A Review
Salza, P., Musmarra, P., & Ferrucci, F. (2018) · Springer eBooks · doi:10.1007/978-981-13-2751-3_2 · Cited by 94
A Decade of Ubiquitous Computing Research in Mental Health
Bardram, J. E., & Matic, A. (2020) · IEEE Pervasive Computing · doi:10.1109/mprv.2019.2925338 · Cited by 77
“Snowballing refers to using the reference list and citations of one selected paper to find related papers.(7) Using references and citations is referred to backward and forward snowballing, respectively.”
MethodBlockchain and cryptocurrencies: A classification and comparison of architecture drivers
Garriga, M., Palma, S. D., Arias, M., Renzis, A. D., Pareschi, R., & Tamburri, D. A. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5992 · Cited by 110
Building Reflexivity Using Service Design Methods
Vink, J., & Koskela-Huotari, K. (2021) · Journal of Service Research · doi:10.1177/10946705211035004 · Cited by 126
Provenance documentation to enable explainable and trustworthy AI: A literature review
Kale, A., Nguyen, T., Harris, F. C., Li, C., Zhang, J., & Ma, X. (2023) · Data Intelligence · doi:10.1162/dint_a_00119 · Cited by 77
“We followed the standard systematic literature review method with backward and forward snowballing strategies (Wohlin, 2014).”
MethodViles, E., Kalemkerian, F., Garza‐Reyes, J. A., Antony, J., & Santos, J. (2022) · Sustainable Production and Consumption · doi:10.1016/j.spc.2022.08.024 · Cited by 120
Duluins, O., & Baret, P. V. (2024) · Nature Food · doi:10.1038/s43016-023-00906-7 · Cited by 115
“Using the guidelines for snow-balling in a systematic literature review 62 , we identified 28 additional articles (Fig.”
BackgroundSoftware Design Smell Detection: a systematic mapping study
Alkharabsheh, K., Crespo, Y., Manso, E., & Taboada, J. A. (2018) · Software Quality Journal · doi:10.1007/s11219-018-9424-8 · Cited by 90
“On the other hand, we adopted the snow-balling technique (Wohlin 2014), which permits the reference list of the final determined studies from the automatic search (electronic databases) to be analyzed.”
BackgroundMethodResultOn the relevance of data science for flight delay research: a systematic review
de Carvalho, L. S., Sternberg, A., Gonçalves, L. M., Cruz, A. B., Soares, J., Brandão, D. N., Carvalho, D., & Ogasawara, E. (2020) · Transport Reviews · doi:10.1080/01441647.2020.1861123 · Cited by 130
“A forward snowballing (Wohlin, 2014) was conducted over journal papers to evaluate”
MethodConsumption in the Circular Economy: Learning from Our Mistakes
Garcia, D. G., Kipnis, E., Vasileiou, E., & Solomon, A. (2021) · Sustainability · doi:10.3390/su13020601 · Cited by 97
“This snowballing approach was particularly useful for identifying studies that use different vocabulary to refer to the same (or similar) concepts, which can be a problem when searching for a given set of keywords [119].”
BackgroundMethodAssessing circularity interventions: a review of EEIOA-based studies
Aguilar‐Hernández, G. A., Sigüenza-Sánchez, C. P., Donati, F., Rodrigues, J. F. D., & Tukker, A. (2018) · Journal of Economic Structures · doi:10.1186/s40008-018-0113-3 · Cited by 103
“We then developed a backwards/forwards snowballing process (Wohlin 2014), identifying additional relevant literature from the citation network.”
MethodIkiz Kaya, D., Pintossi, N., & Dane, G. (2021) · Sustainability · doi:10.3390/su13052479 · Cited by 69
“The ten most-relevant and recent (published in last 5 years) abstracts are then selected for backward snowballing procedure in order to finalize the complete list of included publications [44].”
BackgroundA Systematic Review on Supply Chain Management in Prefabricated House-Building Research
Masood, R., Lim, J. B. P., González, V. A., Roy, K., & Khan, K. I. A. (2022) · Buildings · doi:10.3390/buildings12010040 · Cited by 93
Barth, S., Ionita, D., & Hartel, P. (2022) · ACM Computing Surveys · doi:10.1145/3502288 · Cited by 105
“We then followed a systematic review process [132] using a “snowballing” approach [157] described below to iteratively extend the search query and the sample by examining references (Figure 1).”
MethodForex market forecasting using machine learning: Systematic Literature Review and meta-analysis
Ayitey, M., Appiahene, P., Appiah, O., et al. (2023) · Journal Of Big Data · doi:10.1186/s40537-022-00676-2 · Cited by 75
“In addition, we incorporated the process of systematic reference inclusion, often known as "snowballing," as outlined by [71].”
MethodInfluentialMelchior, C., & Oliveira, M. F. D. A. (2023) · New Media & Society · doi:10.1177/14614448231174224 · Cited by 117
“The next step aimed to maximize the scope of the search; thus, we carried out the snowballing technique with a single interaction (Wohlin, 2014), using the 177 articles found so far.”
MethodA Systematic Review of AI-Driven Educational Assessment in STEM Education
Ouyang, F., Dinh, T. A., & Xu, W. (2023) · Journal for STEM Education Research · doi:10.1007/s41979-023-00112-x · Cited by 111
“After database searching, a snowballing approach was utilized to identify the articles that were not captured by the initial search strings (Wohlin, 2014).”
El-Sharkawy, S., Yamagishi-Eichler, N., & Schmid, K. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.08.015 · Cited by 88
Uncertainty representation in software models: a survey
Troya, J., Moreno, N., Bertoa, M. F., & Vallecillo, A. (2021) · Software & Systems Modeling · doi:10.1007/s10270-020-00842-1 · Cited by 95
“At this stage, we also performed a process of backward snowballing [227], considering the related works described in the selected papers.”
MethodA review of data-driven approaches to predict train delays
Tiong, K. Y., Ma, Z., & Palmqvist, C.-W. (2023) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2023.104027 · Cited by 99
A systematic literature review on quality criteria for agile requirements specifications
Heck, P., & Zaidman, A. (2016) · Software Quality Journal · doi:10.1007/s11219-016-9336-4 · Cited by 91
“We took the remaining 65 papers and applied snowballing according to the guidelines provided by Wohlin (2014).”
MethodSurvey of Approaches for Handling Static Analysis Alarms
Muske, T., & Serebrenik, A. (2016) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2016.25 · Cited by 94
“Snowballing [19], [27] is performed after the KDS due to the following reasons: (a) the keywords used (search strings) in Table I might be incomplete due to several terminologies and their synonyms used in the relevant papers; and (b) more importantly, given a good start set, snowballing approach is found to be more e…”
BackgroundMethodShort text similarity measurement methods: a review
Prakoso, D. W., Abdi, A., & Amrit, C. (2021) · Soft Computing · doi:10.1007/s00500-020-05479-2 · Cited by 93
“This review was conducted based on Kitchen-ham (2004) using snowballing techniques from Wohlin (2014) for performing systematic literature reviews.”
InfluentialModelling the bioeconomy: Emerging approaches to address policy needs
Pyka, A., Cardellini, G., van Meijl, H., & Verkerk, P. J. (2021) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2021.129801 · Cited by 91
“The snowballing procedure (Wohlin, 2014) was used to identify the relevant works to be included.”
MethodModelling in low-code development: a multi-vocal systematic review
Bucaioni, A., Cicchetti, A., & Ciccozzi, F. (2022) · Software & Systems Modeling · doi:10.1007/s10270-021-00964-0 · Cited by 125
“We complemented the initial automatic search of with an exhausting backward and forward snowballing (on Google Scholar) as suggested by Wohlin et al. [88].”
Method
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.