Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 701–800 of 3,934
Haptics in social interaction with agents and avatars in virtual reality: a systematic review
Jacucci, G., Bellucci, A., Ahmed, I., Harjunen, V. J., Spapé, M., & Ravaja, N. (2024) · Virtual Reality · doi:10.1007/s10055-024-01060-6 · Cited by 61
“We identified 45 additional publications from backward and forward snowballing citation searching (Wohlin 2014) from the retrieved publications that matched the screening 2 Unlike author keywords, Keywords Plus® are automatically-gener-ated terms from the titles of cited papers.”
A systematic review on the engineering of software for ubiquitous systems
Guinea, A. S., Nain, G., & Traon, Y. L. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.05.024 · Cited by 49
A Faceted Classification Scheme for Change-Based Industrial Code Review Processes
Baum, T., Liskin, O., Niklas, K., & Schneider, K. A. (2016) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2016.19 · Cited by 61
“We performed “theoretical sampling” to select the intervie-wees: We used our emerging results to choose further participants, for example from a so far little-investigated context.”
MethodClinton‐Lisell, V. (2018) · Psychology Learning & Teaching · doi:10.1177/1475725718799511 · Cited by 49
Mobile and Wearable Sensing Frameworks for mHealth Studies and Applications
Kumar, D., Jeuris, S., Bardram, J. E., & Dragoni, N. (2020) · ACM Transactions on Computing for Healthcare · doi:10.1145/3422158 · Cited by 52
“Specifically, in addition to the proposed search strategy, the backward snowballing technique [90] was applied for finding additional relevant papers.”
MethodGwara, S., Wale, E. Z., Odindo, A. O., & Buckley, C. A. (2021) · Agriculture · doi:10.3390/agriculture11020153 · Cited by 52
“The authors used additional references suggested by some reviewers and hand-searched other articles through backward snowballing from the reference lists of related reviews and included articles [73,81,82].”
MethodQuality measurement in agile and rapid software development: A systematic mapping
López, L., Illa, X. B., Martínez‐Fernández, S., Vollmer, A. M., Behutiye, W., Karhapää, P., Franch, X., Rodríguez, P., & Oivo, M. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111187 · Cited by 75
Smart Contract Languages: A Multivocal Mapping Study
Varela‐Vaca, Á. J., & Quintero, A. M. R. (2021) · ACM Computing Surveys · doi:10.1145/3423166 · Cited by 56
Ziogas, I. A., Evangeliou, A. P., Mylonas, K. S., et al. (2021) · The European Journal of Health Economics · doi:10.1007/s10198-021-01277-1 · Cited by 41
“Reference lists and previously published SRs-MAs were hand-searched for potentially relevant, missed studies utilizing systematic “snowball” procedure guidelines [15].”
MethodA systematic mapping study on automated analysis of privacy policies
Álamo, J. M. D., Guamán, D. S., García, B., & Diez, A. (2022) · Computing · doi:10.1007/s00607-022-01076-3 · Cited by 61
The ethics of self-tracking. A comprehensive review of the literature
Wieczorek, M., Ó’Brolcháin, F., Saghai, Y., & Gordijn, B. (2022) · Ethics & Behavior · doi:10.1080/10508422.2022.2082969 · Cited by 65
“…& Mackinnon, 2018; Sanders, 2017; Sharon, 2017; Sharon & Zandbergen, 2017; Till, 2018; Toner, 2018) In order to extract relevant articles from a greater number of disciplines and broaden the results of the database search, we engaged in backwards snowballing (Jalali & Wohlin, 2012; Wohlin, 2014).”
BackgroundLv, L., Zhong, B., & Liu, X. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11518-2 · Cited by 59
“Accordingly, literature snowballing has been adopted to track the papers that may be ignored initially (Wohlin, 2014; Authors, 2018).”
Backgroundvan Dreven, J., Boeva, V., Abghari, S., Grahn, H., Koussa, J. A., & Motoasca, E. (2023) · Electronics · doi:10.3390/electronics12061448 · Cited by 58
Wens, F. S. P. L., Hulsker, C. C. C., Fiocco, M. F., Zsíros, J., Smetsers, S. E., de Krijger, R. R., van der Steeg, A. F. W., Zweemer, R. P., Baas, I. O., Roes, E. M., et al. (2023) · Cancers · doi:10.3390/cancers15153794 · Cited by 56
“Finally, the Guidelines for Snowballing in Systematic Literature were used to conduct a manual search of references [13].”
MethodWilliams, T. G., Bui, S., Conti, C., Debonne, N., Levers, C., Swart, R., & Verburg, P. H. (2023) · Global Environmental Change · doi:10.1016/j.gloenvcha.2023.102746 · Cited by 47
Yeung, R. C. Y., Yeung, C. H., Sun, D., & Looi, C. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.104999 · Cited by 68
A review of digital assistants in production and logistics: applications, benefits, and challenges
Zheng, T., Grosse, E. H., Morana, S., & Glock, C. H. (2024) · International Journal of Production Research · doi:10.1080/00207543.2024.2330631 · Cited by 54
“Finally, a snowball search using the reference lists of the sampled papers or citations of the sampled papers was conducted to identify additional papers (Wohlin 2014).”
Semantic Web Technologies for Explainable Machine Learning Models: A Literature Review.
Seeliger, A., Pfaff, M., & Krcmar, H. (2019) · Cited by 81
How far are we from reproducible research on code smell detection? A systematic literature review
Lewowski, T., & Madeyski, L. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106783 · Cited by 50
How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40
“We searched Google Scholar with the keywords "sensemaking data analysis" from which we iteratively snowball sampled [117].”
MethodA Systematic Literature Review on Client Selection in Federated Learning
Smestad, C., & Li, J. (2023) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3593434.3593438 · Cited by 60
“To summarize the state of the art of client selection of FL and to answer our research questions, we performed a systematic literature review based upon the guidelines [22] and [53].”
InfluentialReyes, S. M., Brockway, M. L., McDermid, J. M., Chan, D., Granger, M. J., Refvik, R., Sidhu, K. K., Musse, S., Monnin, C., Lotoski, L. C., et al. (2023) · Advances in Nutrition · doi:10.1016/j.advnut.2023.06.005 · Cited by 50
Profile model for management of sustainability integration in engineering design requirements
Watz, M. S., & Hallstedt, S. I. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.119155 · Cited by 40
A systematic literature review on semantic web enabled software testing
Dadkhah, M., Araban, S., & Paydar, S. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.110485 · Cited by 54
“identify new relevant publications [28].”
Yussupov, V., Soldani, J., Breitenbücher, U., Brogi, A., & Leymann, F. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110906 · Cited by 57
“As additional means to identify relevant literature, we applied the snowballing technique [30].”
MethodThe Role of Project Manager in Agile Software Teams: A Systematic Literature Review
Gandomani, T. J., Tavakoli, Z., Zulzalil, H. B., & Farsani, H. K. (2020) · IEEE Access · doi:10.1109/access.2020.3004450 · Cited by 60
“At the same time, parallel snowballing technique was used together with database search to achieve high-quality results [29], [30].”
MethodA systematic review of UAV applications to education
Shadiev, R., & Yi, S. (2022) · Interactive Learning Environments · doi:10.1080/10494820.2022.2028858 · Cited by 49
“The keyword search approach is the process of fi nding related articles by entering keywords into search engines and the snowballing approach is a search method that uses references or citations of papers to determine additional papers (Wohlin, 2014).”
MethodA mapping study on documentation in Continuous Software Development
Theunissen, T., van Heesch, U., & Avgeriou, P. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2021.106733 · Cited by 40
“2) [427].”
BackgroundMethodInfluentialvan Lotringen, C. M., Jeken, L., Westerhof, G. J., ten Klooster, P. M., Kelders, S. M., & Noordzij, M. L. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.689750 · Cited by 47
Cucchi, M., Mariani, S., Piero, M. E. D., Ravaux, J. M., Kawczynski, M. J., Mauro, M. D., Shkurka, E., Hoskote, A. U., & Lorusso, R. (2022) · Perfusion · doi:10.1177/02676591221096078 · Cited by 44
“Reference lists of assessed full texts were screened for further relevant studies.(13)”
BackgroundEvaluation of Marker Materials and Spectroscopic Methods for Tracer-Based Sorting of Plastic Wastes
Olscher, C., Jandric, A., Zafiu, C., & Part, F. (2022) · Polymers · doi:10.3390/polym14153074 · Cited by 46
“Aside from the resulting literature found by the search engines, a snowballing procedure was also applied to find more relevant marker materials and information on their application [40].”
MethodExplainable software systems: from requirements analysis to system evaluation
Chazette, L., Brunotte, W., & Speith, T. (2022) · Requirements Engineering · doi:10.1007/s00766-022-00393-5 · Cited by 62
“The snowballing process includes backward and forward snowballing as described by Wohlin [58].”
MethodEthical Aspects of ChatGPT in Software Engineering Research
Akbar, M. A., Khan, A. A., & Liang, P. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3318183 · Cited by 49
“Furthermore, we employed the snowballing data sampling approach to collect potential literature material related to the study objective [20].”
InfluentialArchitecting ML-enabled systems: Challenges, best practices, and design decisions
Nazir, R., Bucaioni, A., & Pelliccione, P. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111860 · Cited by 57
Forms of participatory futuring for urban sustainability: A systematic review
Neuhoff, R., Simeone, L., & Laursen, L. L. H. (2023) · Futures · doi:10.1016/j.futures.2023.103268 · Cited by 54
Exploring time series models for landslide prediction: a literature review
Ebrahim, K. M. P., Fares, A. I., Faris, N., & Zayed, T. M. (2024) · Geoenvironmental Disasters · doi:10.1186/s40677-024-00288-3 · Cited by 54
“To broaden the scope of the search, the backward and forward snowballing method (Wohlin 2014) was employed, resulting in the discovery of additional relevant articles beyond those identified through the Scopus search.”
MethodBukowski, L. A., & Werbińska-Wojciechowska, S. (2025) · Sensors · doi:10.3390/s25165100 · Cited by 68
A Survey on Software Release Planning Models
Ameller, D., Farré, C., Franch, X., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49094-6_4 · Cited by 39
Pricing of Data Products in Data Marketplaces
Fricker, S. A., & Maksimov, Y. V. (2017) · Lecture notes in business information processing · doi:10.1007/978-3-319-69191-6_4 · Cited by 51
“The search efficiency of 6% is a figure that can be found in other literature reviews [8].”
BackgroundMethodBlattgerste, J., Renner, P., & Pfeiffer, T. (2019) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3316782.3316789 · Cited by 54
“[73] and combining the results with our already performed database search, 26 additional publications (9 for learning, 9 for action assistance, and 8 for cognitive rehabilitation) that satisfy our inclusion criteria were identified, resulting in 67 publications.”
BackgroundMethodSmart-troubleshooting connected devices: Concept, challenges and opportunities
Caporuscio, M., Flammini, F., Khakpour, N., Singh, P., & Thornadtsson, J. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2019.09.004 · Cited by 43
The Landscape of Envisioning and Speculative Design Methods for Sustainable Mobility Futures
Sustar, H., Mladenović, M. N., & Givoni, M. (2020) · Sustainability · doi:10.3390/su12062447 · Cited by 48
“In the second, screening stage, a snowballing procedure was used to increase the pool of references [59], including both backward- and forward-snowballing [60].”
MethodA system boundary-based critical review on crane selection in building construction
Wang, R. D., Zayed, T. M., Pan, W., Zheng, S., & Tariq, S. (2020) · Automation in Construction · doi:10.1016/j.autcon.2020.103520 · Cited by 37
Building a Theoretical Framework for Corporate Sustainability
Sánchez-Planelles, J., Segarra‐Oña, M., & Peiró‐Signes, Á. (2020) · Sustainability · doi:10.3390/su13010273 · Cited by 62
Singh, R. K., Singh, A., Zander, K. K., Mathew, S., & Kumar, A. (2021) · Journal of Environmental Management · doi:10.1016/j.jenvman.2020.111679 · Cited by 51
Emotions and programming learning: systematic mapping
Coto, M., Mora, S., Grass, B. E., & Murillo-Morera, J. (2021) · Computer Science Education · doi:10.1080/08993408.2021.1920816 · Cited by 48
“To identify additional relevant studies in addition to those retrieved with the search string, a snowball technique was used (Wohlin, 2014) to identify new papers from the reference lists of the initially selected papers, for which the same inclusion and exclusion criteria were applied.”
MethodZhao, F., & Schuchardt, A. M. (2021) · International Journal of STEM Education · doi:10.1186/s40594-020-00264-x · Cited by 61
“…of an initial set of manuscripts, a backward screening on the reference lists of the starting set of manuscripts, a forward screening on the publications citing the starting set of manuscripts, and iteration of backward and forward screening on the included publications (Wohlin, 2014).”
BackgroundMethodOperationalizing Human Values in Software Engineering: A Survey
Shahin, M., Hussain, W., Nurwidyantoro, A., Perera, H., Shams, R. A., Grundy, J., & Whittle, J. (2022) · IEEE Access · doi:10.1109/access.2022.3190975 · Cited by 59
“Following the guidelines proposed by Wohlin [46], the first author conducted the backward snowballing in several iterations until no new papers were found.”
MethodCybersecurity Testing for Automotive Domain: A Survey
Luo, F., Zhang, X., Yang, Z., Jiang, Y., Wang, J., Wu, M., & Feng, W. (2022) · Sensors · doi:10.3390/s22239211 · Cited by 46
“Finally, we identified new papers through backward snowballing, a method for including new papers by examining the title, publication venue and authors in the reference list of the selected paper [14].”
Artificial intelligence-driven serious games in health care: scoping review
Abd-alrazaq, A., Abuelezz, I., Hassan, A., AlSammarraie, A., Alhuwail, D., Irshaidat, S., Abu Serhan, H., Ahmed, A., Alabed Alrazak, S., & Househ, M. (2022) · JMIR Serious Games · doi:10.2196/39840 · Cited by 39
“To identify further studies, we screened the reference lists of the included studies and relevant reviews (ie, backward reference list checking), and we checked the studies that cited the included studies (ie, forward reference list checking) [38].”
MethodFoidl, H., Felderer, M., & Ramler, R. (2022) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3522664.3528590 · Cited by 51
“After conducting forward and backward snowballing [62] on the identified sources, we extracted all latent data issues in a spreadsheet.”
Fifty years of artisan entrepreneurship: a systematic literature review
Hasanah, U., Sukoco, B. M., Supriharyanti, E., & Wu, W. (2023) · Journal of Innovation and Entrepreneurship · doi:10.1186/s13731-023-00308-w · Cited by 68
“The second stage involved searching and filtering the literature by adopting a snowballing procedure (Wohlin, 2014), such as developing a start set and iteration by applying criteria to determine the appropriate article as follows:”
MethodChueca, J., Verón, J., Font, J., Pérez, F., & Cetina, C. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107330 · Cited by 63
Newbutt, N. A., Glaser, N. J., Francois, M.-S., Schmidt, M., & Cobb, S. V. G. (2023) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-023-06130-3 · Cited by 48
“In stage two, we manually searched the reference lists of all identified relevant articles using ancestral searching (reviewing references in the literature of our included articles), which reduced the risk of missing relevant studies (Wohlin, 2014).”
MethodDesigning interpretable deep learning applications for functional genomics: a quantitative analysis
van Hilten, A., Katz, S., Saccenti, E., Niessen, W., & Roshchupkin, G. (2024) · Briefings Bioinform. · doi:10.1093/bib/bbae449 · Cited by 35
“Additionally, we included relevant articles identified by the snowballing procedure [36], resulting in a total of 123 research articles for our analysis.”
MethodHabiba, U.-E., Haug, M., Bogner, J., & Wagner, S. (2024) · Requirements Engineering · doi:10.1007/s00766-024-00432-3 · Cited by 95
“Snowballing: Following the first iteration of extractions, we applied forward and backward snowballing according to the guidelines by Wohlin [25].”
Matalonga, S., Rodrigues, F. F., & Travassos, G. H. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.05.048 · Cited by 43
“Finally, to further mitigate this threat, we performed Snowballing [48] from the selected primary studies to identify any relevant technical literature which they were referencing, and for some reason, we might have missed.”
BackgroundMethodInfluentialInvestigating transparency regarding ecoinvent users’ system model choices
Saade, M. R. M., da Silva, V. G., da Silva, M. G., Ugaya, C. M. L., Lasvaux, S., Passer, A., & Habert, G. (2018) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-018-1509-x · Cited by 38
Wieser, A. A., Scherz, M., Maier, S., Passer, A., & Kreiner, H. (2019) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/323/1/012177 · Cited by 52
A classification framework for automated control code generation in industrial automation
Koziolek, H., Burger, A., Platenius‐Mohr, M., et al. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110575 · Cited by 38
“We applied a forward/backward reference search [53] on the identified approaches to increase literature coverage.”
MethodA Systematic Review of Rapid Prototyping Tools for Augmented Reality
Freitas, G., Pinho, M. S., Silveira, M. S., & Maurer, F. (2020) · Symposium on Virtual and Augmented Reality (SVR) · doi:10.1109/svr51698.2020.00041 · Cited by 48
“Finally, we conducted a forward and backward snowballing4, following the guidelines proposed by Wohlin [53],”
MethodGarrido, L. C., Moya, A. V., & Morancho, M. V. (2021) · Journal of New Approaches in Educational Research · doi:10.7821/naer.2021.1.591 · Cited by 56
“The Snowball technique consists of identifying relevant references cited in the selected papers in order to increase the number of sources revised (Wnuk & Garrepalli, 2018; Wohlin, 2014).”
Testing Microservices Architecture-Based Applications: A Systematic Mapping Study
Waseem, M., Liang, P., Márquez, G., & Salle, A. D. (2020) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec51365.2020.00020 · Cited by 69
“Phase II - Snowballing : Further relevant studies are identified by applying the snowballing procedure [19].”
Neurophysiological Measurements in Higher Education: A Systematic Literature Review
Darvishi, A., Khosravi, H., Sadiq, S., & Weber, B. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00256-0 · Cited by 37
“Thesnowballing approach of Wohlin (2014), was applied to the remaining 57 papers.”
MethodCustomer experience research: intellectual structure and future research opportunities
Silva, J. H. O., Mendes, G. H. D. S., Miguel, P. A. C., Amorim, M. P. C., & Teixeira, J. (2021) · Journal of Service Theory and Practice · doi:10.1108/jstp-08-2020-0193 · Cited by 53
Systematic analysis of healthcare big data analytics for efficient care and disease diagnosing
Khan, S., Khan, H. U., & Nazir, S. (2022) · Scientific Reports · doi:10.1038/s41598-022-26090-5 · Cited by 59
“• Snowballing To extract each relevant primary article snowballing is applied in the proposed research work 33 .”
MethodSoofi, A. A., & Uddin, M. (2019) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2019.0100312 · Cited by 37
“For strengthen the search related to relevant articles, backward snowballing guidelines presented in [24] was also adopted to find out cited papers in selected articles that meet our inclusion criteria; this was done to make sure that no articles related to our research questions were left to considered.”
MethodCognition in Software Engineering: A Taxonomy and Survey of a Half-Century of Research
Fagerholm, F., Felderer, M., Fucci, D., Unterkalmsteiner, M., Mărculescu, B., Martini, M., Tengberg, L. G. W., FELDT, R. H., Lehtelä, B., Nagyváradi, B., et al. (2022) · ACM Computing Surveys · doi:10.1145/3508359 · Cited by 62
“To answer RQ2, we conducted a literature survey driven by keyword search and backward snowball sampling [61] as described in Sections 3.”
MethodModern Code Reviews—Survey of Literature and Practice
Badampudi, D., Unterkalmsteiner, M., & Britto, R. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3585004 · Cited by 75
“[39]), are adequate for quantitative research.”
Multidisciplinary perspectives on rebound effects in sustainability: A systematic review
Andrew, E. M., & Pigosso, D. C. A. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.143366 · Cited by 55
On the application of image augmentation for plant disease detection: A systematic literature review
Antwi, K., Bennin, K. E., Asiedu, D. K. P., & Tekinerdoğan, B. (2024) · Smart Agricultural Technology · doi:10.1016/j.atech.2024.100590 · Cited by 66
Calderón, A., Ruiz, M., & O’Connor, R. V. (2018) · Computer Standards & Interfaces · doi:10.1016/j.csi.2018.04.012 · Cited by 44
Landscaping systematic mapping studies in software engineering: A tertiary study
Khan, M. U., Sherin, S., Iqbal, M. Z., & Zahid, R. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.12.018 · Cited by 51
“to conduct the process of snowballing [32].”
A Systematic Mapping Study on Engineering Function-as-a-Service Platforms and Tools
Yussupov, V., Breitenbücher, U., Leymann, F., & Wurster, M. (2019) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1145/3344341.3368803 · Cited by 51
“In the next phase, to increase the set of relevant literature we applied a complementary activity referred to as snow-balling technique [29].”
MethodHealthcare professionals' trust in patients: A review of the empirical and theoretical literatures
Sousa‐Duarte, F., Brown, P. R., & Mendes, A. M. (2020) · Sociology Compass · doi:10.1111/soc4.12828 · Cited by 49
“We ran additional searches on Google Scholar to avoid risk of ‘publisher bias’ caused by searches in specific publishers' databases (Wohlin, 2014, p. 5).”
BackgroundMethodInfluentialMohamed, M. A., Kardaş, G., & Challenger, M. (2021) · IEEE Access · doi:10.1109/access.2021.3068358 · Cited by 51
“Conforming to the snowballing guidelines given in [42], the forward snowballing process was accomplished in this study by determining other papers citing any of the primary studies.”
BackgroundMethodCode Smells Detection Using Artificial Intelligence Techniques: A Business-Driven Systematic Review
Lewowski, T., & Madeyski, L. (2021) · Studies in systems, decision and control · doi:10.1007/978-3-030-77916-0_12 · Cited by 56
Ahamad, M. A., Arifin, K., Abas, A., Mahfudz, M., Basir-Cyio, M., Khairil, M., Ali, M. N., Lampe, I., & Samad, M. A. (2022) · Sustainability · doi:10.3390/su14137523 · Cited by 55
Reviewer assignment algorithms for peer review automation: A survey
Zhao, X., & Zhang, Y. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103028 · Cited by 53
Obstacles to strategy implementation and success factors: A review of empirical literature
Vigfússon, K., Jóhannsdóttir, L., & Ólafsson, S. (2021) · Strategic Management · doi:10.5937/straman2102012v · Cited by 97
Alfaro-Díaz, C., Svavarsdottir, E. K., Esandi, N., Klinke, M. E., & Canga-Armayor, A. (2022) · Journal of Family Nursing · doi:10.1177/10748407211068816 · Cited by 36
“In the third and final phase, the backward and forward snowballing technique was conducted to search studies in reference lists of identified papers (Wohlin, 2014).”
MethodPaschalidou, E., Fafet, C., & Milios, L. (2022) · Heritage · doi:10.3390/heritage5020058 · Cited by 58
“With the snowball method, additional literature was sourced from the bibliography found in the relevant articles [53].”
MethodAssessing Writing Motivation: a Systematic Review of K-5 Students' Self-Reports
Alves-Wold, A., Walgermo, B. R., McTigue, E. M., & Uppstad, P. H. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09732-6 · Cited by 55
“For this search step, we used both Scopus and Google Scholar to identify all the papers who had cited the 51 included studies in our review, and screened these references as described by Wohlin (2014).”
MethodThe role of global health partnerships in vaccine equity: A scoping review
Nunes, C., McKee, M., & Howard, N. (2024) · PLOS Global Public Health · doi:10.1371/journal.pgph.0002834 · Cited by 41
“Lastly, we backward-searched references of eligible articles [17].”
MethodRazzaq, A., Buckley, J., Lai, Q., Yu, T., & Botterweck, G. (2025) · ACM Computing Surveys · doi:10.1145/3687299 · Cited by 36
“(3) Snowballing: To further increase the scope of the paper, we performed one iteration of snow-balling, as per the guidelines presented by Wohlin [27].”
Lackner, T., Hermann, J., Kühn, C., & Palm, D. (2024) · Procedia CIRP · doi:10.1016/j.procir.2024.10.187 · Cited by 80
Agentic AI Frameworks in SMMEs: A Systematic Literature Review of Ecosystemic Interconnected Agents
Olujimi, P. A., Owolawi, P. A., Mogase, R. C., & Wyk, E. V. (2025) · AI · doi:10.3390/ai6060123 · Cited by 36
Quality in model-driven engineering: a tertiary study
Goulão, M., Amaral, V., & Mernik, M. (2016) · Software Quality Journal · doi:10.1007/s11219-016-9324-8 · Cited by 36
“…to be potentially included, which may then be complemented with other strategies, as suggested in (Kitchenham and Charters 2007), (Giraldo et al 2015) use the backward snowball approach, which was recently evaluated as an effective alternative for performing literature reviews in (Wohlin 2014).”
Garousi, V., & Felderer, M. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084238 · Cited by 48
“By being a ‘guideline’ paper, similar to [1, 10, 12, 28], the main contributions of this paper are a synthesized list of challenges usually faced during SLRs’ data extraction phase and the corresponding solutions (guidelines), synthesized from two sources: (1) the pool of 16 SLR studies in which…”
BackgroundMethodA systematic review on the use of Definition of Done on agile software development projects
Silva, A., Araújo, T., Nunes, J., Perkusich, M., Dilorenzo, E., de Almeida, H. O., & Perkusich, Â. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084262 · Cited by 56
“We decided not to follow the guidelines for snowballing presented in Wohlin [16], in which it is recommended that the nal inclusion of a paper should be done based on the full paper, because our research questions are too speci c.”
BackgroundMethodA Literature Study on Privacy Patterns Research
Lenhard, J., Fritsch, L., & Herold, S. (2017) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2017.28 · Cited by 47
“II-B), we applied backward and forward snowballing based on the articles considered relevant for the study [18].”
MethodADDITIVE MANUFACTURING FROM A STRATEGIC SUSTAINABILITY PERSPECTIVE
Villamil, C., Nylander, J. W., Hallstedt, S. I., Schulte, J., & Watz, M. S. (2018) · Proceedings of the ... International Design Conference/Design ... · doi:10.21278/idc.2018.0353 · Cited by 43
Proxy Measures for Simplified Environmental Assessment of Manufactured Nanomaterials
Arvidsson, R., Baun, A., Furberg, A., Hansen, S. F., & Molander, S. (2018) · Environmental Science & Technology · doi:10.1021/acs.est.8b05405 · Cited by 39
Multilayered review of safety approaches for machine learning-based systems in the days of AI
Dey, S., & Lee, S.-W. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110941 · Cited by 53
“After getting a moderate number of search results 285 from these data sources, we followed the forward and backward snowballing approach [43] to find new relevant papers (from the references and the citation lists) in Phase-2.”
MethodA Review on C3I Systems’ Security: Vulnerabilities, Attacks, and Countermeasures
Ahmad, H., Dharmadasa, I., Ullah, F., & Babar, M. A. (2023) · ACM Computing Surveys · doi:10.1145/3558001 · Cited by 61
“• Snowballing: According to the snowballing technique reported in [27], we explored the references of the selected 63 studies, and retrieved 27 potentially relevant papers.”
MethodInfluentialTopping, C., Dwyer, A. C., Michalec, O., Craggs, B., & Rashid, A. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102324 · Cited by 59
On preserving the behavior in software refactoring: A systematic mapping study
AlOmar, E. A., Mkaouer, M. W., Newman, C. D., & Ouni, A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106675 · Cited by 59
“To conduct this systematic mapping study, we followed established guidelines for SLR and SLM studies in SE [4, 5, 6].”
MethodInfluentialA Systematic Mapping Study of Predictive Maintenance in SMEs
Khan, M., Ahmad, A., Sobieczky, F., Pichler, M., Moser, B., & Bukovský, I. (2022) · IEEE Access · doi:10.1109/access.2022.3200694 · Cited by 45
“Snowballing follows the guidelines of [58], [59] as backward and forward snowballing.”
MethodIncorporating user preferences in search-based software engineering: A systematic mapping study
Ferreira, T. N., Vergílio, S. R., & Souza, J. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.05.003 · Cited by 45
Agentic AI Frameworks in SMMEs: A Systematic Literature Review of Ecosystemic Interconnected Agents
Olujimi, P. A., Owolawi, P. A., Mogase, R. C., & Wyk, E. V. (2025) · doi:10.20944/preprints202504.1797.v1 · Cited by 36
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.