Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 1,101–1,200 of 3,934
Overeem, M., Spoor, M., Jansen, S., & Brinkkemper, S. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.110970 · Cited by 42
“We executed interview snowballing , a process similar to snowballing in systematic literature studies Wohlin (2014): we explicitly asked every interviewee for further references.”
MethodChen, D., Au, W. Y., van Ewijk, S., Roy, A. D., & Stegemann, J. A. (2021) · Journal of Hazardous Materials · doi:10.1016/j.jhazmat.2021.125676 · Cited by 25
Ziogas, I. A., Esagian, S. M., Giannis, D., Hayat, M. H., Kosmidis, D., Matsuoka, L. K., Montenovo, M. I., Tsoulfas, G., Geller, D. A., & Alexopoulos, S. P. (2021) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2021.03.052 · Cited by 23
Morelli, N. M., & Villodas, M. T. (2021) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-021-00374-0 · Cited by 30
“Next, a snowball technique was applied following guidelines from Wohlin (2014) whereby reference lists of selected articles were screened “backward” (i.e., references of included studies are studied to identify more articles) and “forward” (i.e., articles citing the selected papers are evaluated).”
MethodHadid, M., Elomri, A., ElMekkawy, T. Y., Kerbache, L., Omri, A. E., Omri, H. E., Taha, R. Y., Hamad, A. A., & Thani, M. H. J. A. (2022) · Health Care Management Science · doi:10.1007/s10729-021-09585-x · Cited by 26
“The backward and forward snowballing approaches were used to reach relevant papers not found by the search query [129].”
MethodLiving Labs in Social Housing Upgrades: Process, Challenges and Recommendations
Bridi, M. E., Soliman-Junior, J., Granja, A. D., et al. (2022) · Sustainability · doi:10.3390/su14052595 · Cited by 23
Microclimate of Air Cavities in Ventilated Roof and Façade Systems in Nordic Climates
Ingebretsen, S. B., Andenæs, E., & Kvande, T. (2022) · Buildings · doi:10.3390/buildings12050683 · Cited by 23
Analysis of ontologies and policy languages to represent information flows in GDPR
Esteves, B. G. C., & Rodríguez-Doncel, V. (2022) · Semantic Web · doi:10.3233/sw-223009 · Cited by 51
“Following this approach, the search for academic publications to be included in this review, and other published documentation as this is a distinctive feature of an integrative review, was performed according to the snowballing procedure [98].”
MethodSentiment analysis techniques, challenges, and opportunities: Urdu language-based analytical study
Liaqat, M. I., Awais, M., Shoaib, M., Khurshid, S. K., & Shamseldin, M. Α. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1032 · Cited by 27
“The references of the selected articles were also explored to backtrack the essential articles, and several articles are included by adopting the snowballing ( Wohlin, 2014 ) technique.”
BackgroundFailure and fault classification for smart grids
Krivohlava, Z., Chren, S., & Rossi, B. (2022) · Energy Informatics · doi:10.1186/s42162-022-00218-3 · Cited by 32
“…we needed very specific types of papers to collect faults and failures, we adopted a specific search strategy: collecting first a set of core relevant papers and then looking at the referenced papers that could provide more relevant results [(so-called snowballing in SLR terminology (Wohlin 2014)].”
MethodAn evidence-based roadmap for IoT software systems engineering
Motta, R. C., de Oliveira, K. M., & Travassos, G. H. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111680 · Cited by 30
Future air transportation and digital work at airports – Review and developments
Thums, J., Künzel, L., Klumpp, M., Bardmann, M.-M., & Ruiner, C. (2023) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2023.100808 · Cited by 43
Burnout among healthcare professionals in Qatar: A systematic review
Albazoon, F., Khogali, F., Burjaq, R., Chandra, P., Alabdulla, M., Abdulaziz, M., & Hammoudeh, S. (2023) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2023.103601 · Cited by 29
Safety Testing of Automated Driving Systems: A Literature Review
Khan, F., Falco, M., Anwar, H., & Pfahl, D. (2023) · IEEE Access · doi:10.1109/access.2023.3327918 · Cited by 29
“Our method to identify relevant papers uses the backward snowballing procedure introduced by Wohlin [19].”
MethodSystematic Review on Requirements Engineering in Quantum Computing: Insights and Future Directions
Sepúlveda, S., Cravero, A., Fonseca, G. M., & Antonelli, L. (2024) · Electronics · doi:10.3390/electronics13152989 · Cited by 38
Customer Involvement in Continuous Deployment: A Systematic Literature Review
Yaman, S. G., Sauvola, T., Riungu-Kalliosaari, L., Hokkanen, L., Kuvaja, P., Oivo, M., & Männistö, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-30282-9_18 · Cited by 34
“New papers were added to the primary study collection by performing backward snowball sampling [12] and tracking key researchers’ work in the field.”
BackgroundStepanovic, S., & Mettler, T. (2018) · IRIS · Cited by 46
Pendergraft, J. G., Carter, D. R., Tseng, S., Landon, L. B., Slack, K. J., & Shuffler, M. L. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01633 · Cited by 26
“Our collection of archival documents progressed in a series of three steps and leveraged an adapted snowballing review technique (Wohlin, 2014).”
MethodJohnson, M., Albizri, A., & Jain, R. (2020) · Decision Sciences Journal of Innovative Education · doi:10.1111/dsji.12195 · Cited by 30
Zaldo-Aubanell, Q., Serra, I., Sardanyés, J., Alsedà, L., & Maneja, R. (2020) · Environmental Research · doi:10.1016/j.envres.2020.110578 · Cited by 26
Garcia, E. S., Huysveld, S., Nhu, T. T., Meester, S. D., & Dewulf, J. P. (2023) · Waste Management · doi:10.1016/j.wasman.2023.08.032 · Cited by 32
Loufopoulos, G., Tasoudis, P., Koudounas, G., Zoupas, I., Madouros, N., Sá, M. P., Karkos, C. D., Giannopoulos, S., & Tassiopoulos, A. K. (2023) · Journal of Endovascular Therapy · doi:10.1177/15266028231204805 · Cited by 22
“20 The Covidence reference and article manager software was used for all stages of the database search and study selection.”
Gurmu, A. T., & Mahmood, M. N. (2024) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-13794 · Cited by 22
Emergence of eye-tracking in architectural research: a review of studies 1976–2021
Aalto, P., & Steinert, M. (2024) · Architectural Science Review · doi:10.1080/00038628.2024.2382796 · Cited by 35
“This includes a systematic database search (Callahan 2014) followed by a snowballing approach (Badampudi, Wohlin, and Petersen 2015; Felizardo et al. 2016; Wohlin2014 ).”
Linking Disparate Strands: A Critical Review of the Relationship Between Creativity and Education
Rawlings, B. S., & Cutting, S. J. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09973-z · Cited by 41
“After identifying relevant literature, we then further identified relevant research using the reference lists and citations (snowballing) from manuscripts identified in our Scopus search (Wohlin, 2014).”
From Traditional to Digital: The Evolution of Business Models in Hospitality Through Platforms
Zeqiri, A. (2024) · Platforms · doi:10.3390/platforms2040015 · Cited by 59
“[45] notes that snowballing can effectively extend a systematic literature study, as new studies often reference previously identified relevant research.”
Space Economy and Sustainability: A Systematic Review
Valente, M., Caviggioli, F., & Agostini, L. (2025) · Sustainable Development · doi:10.1002/sd.3383 · Cited by 27
AI Techniques in the Microservices Life-Cycle: a Systematic Mapping Study
Moreschini, S., Pour, S. M., Lanese, I., Balouek‐Thomert, D., Bogner, J., Li, X., Pecorelli, F., Soldani, J., Truyen, E., & Taibi, D. (2025) · Computing · doi:10.1007/s00607-025-01432-z · Cited by 69
“Snowballing: We performed the snowballing process [32], considering all the references presented in the retrieved papers and evaluating all the papers referencing the retrieved ones.”
Guidelines for Architecting Android Apps: A Mixed-Method Empirical Study
Verdecchia, R., Malavolta, I., & Lago, P. (2019) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa.2019.00023 · Cited by 32
“We opt for such digital library as (i) its adoption constitutes a sound choice to determine the initial set of literature for snowballing processes [9], (ii) from a preliminary execution of the search query it results to be more inclusive w.r.t. Scopus and IEEE Explore, and (iii) the results of the…”
MethodResultA review of visual ethnography: Radiography viewed through a different lens
O’Regan, T., Robinson, L., Newton-Hughes, A., & Strudwick, R. M. (2019) · Radiography · doi:10.1016/j.radi.2019.06.007 · Cited by 25
Hodrien, A., & Fernando, T. (2021) · University of Salford Institutional Repository (University of Salford) · Cited by 39
A Comprehensive Exploration of Languages for Parallel Computing
Ciccozzi, F., Addazi, L., Asadollah, S. A., Lisper, B., Masud, A. N., & Mubeen, S. (2022) · ACM Computing Surveys · doi:10.1145/3485008 · Cited by 31
“We performed a closed recursive backward and forward snowballing activity [253].”
MethodAn ontology to represent geospatial data to support building renovation
Daneshfar, M., Hartmann, T., & Rabe, J. (2022) · Advanced Engineering Informatics · doi:10.1016/j.aei.2022.101591 · Cited by 23
Considering temporal aspects in recommender systems: a survey
Bogina, V., Kuflik, T., Jannach, D., Bieliková, M., Kompan, M., & Trattner, C. (2022) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-022-09335-w · Cited by 29
“After identifying and scrutinizing relevant papers, we applied a snowballing procedure, where the papers’ references list as well as the papers citing them were examined and considered (Wohlin 2014).”
MethodConsumer or Patient Determinants of Hospital Brand Equity—A Systematic Literature Review
Górska-Warsewicz, H. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159026 · Cited by 67
“Since a solution was applied due to different research methods, different statistical tools, and different research scales, we could not use methods typical of metaanalysis [87,88].”
MethodLi, Z. Z., Joseph, K. L., Yu, J., & Gašević, D. (2022) · IEEE International Conference on Blockchain (Blockchain) · doi:10.1109/blockchain55522.2022.00021 · Cited by 36
“This paper uses snowballing approach in software engineering [28], built on a ”start set” [28] of recent systematic literature review studies [5], [6], [8], to select representative studies.”
MethodHansen, R. N., Rasmussen, F. N., Ryberg, M. W., & Birgisdóttir, H. (2022) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-022-02126-w · Cited by 31
Berti, S., Grazia, V., & Molinari, L. (2023) · Educational Psychology Review · doi:10.1007/s10648-023-09773-x · Cited by 79
Integration of convolutional and adversarial networks into building design: A review
Parente, J. M., Rodrigues, E. M. D. S., Rangel, B., & Martins, J. P. (2023) · Journal of Building Engineering · doi:10.1016/j.jobe.2023.107155 · Cited by 29
“Lastly, the snowballing method was used, an iterative process with two complementary parts: backward and forward [43].”
MethodDesign Patterns for AI-based Systems: A Multivocal Literature Review and Pattern Repository
Heiland, L., Hauser, M., & Bogner, J. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00035 · Cited by 30
“To extend the initial search results, we performed one round of forward and backward snowballing following Wohlin [34].”
MethodAdaptive user interfaces in systems targeting chronic disease: a systematic literature review
Wang, W., Khalajzadeh, H., Grundy, J., Madugalla, A. K., McIntosh, J., & Obie, H. O. (2023) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09384-9 · Cited by 43
3D Cadastral Database Systems—A Systematic Literature Review
Shahidinejad, J., Kalantari, M., & Rajabifard, A. (2024) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi13010030 · Cited by 32
SLAM for Visually Impaired People: A Survey
Bamdad, M., Scaramuzza, D., & Darvishy, A. (2024) · IEEE Access · doi:10.1109/access.2024.3454571 · Cited by 43
“To provide a broader range of relevant studies, we included papers on the forward snowballing process [55].”
BackgroundAn empirical study of portfolio management and Kanban in agile and lean software companies
Ahmad, M. O., Lwakatare, L. E., Kuvaja, P., Oivo, M., & Markkula, J. (2016) · Journal of Software Evolution and Process · doi:10.1002/smr.1834 · Cited by 27
“According to Wohlin, 40 a good starting point may be identified by using the Google Scholar search engine, which helps to avoid any bias toward researchers or any specific publisher.”
BackgroundMethodInfluentialFramework for Prioritization of Open Data Publication: An Application to Smart Cities
Prieto, Á. E., Mazón, J.-N., & Lozano-Tello, A. (2019) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2019.2893016 · Cited by 27
“In order to identify relevant related work on how open data is published by governments, we have considered a backward and forward snowball method [26] by navigating citations and references from a starting set of research papers in an iterative fashion.”
MethodA systematic mapping study on higher order mutation testing
Lima, J. P., & Vergílio, S. R. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.04.031 · Cited by 26
Madsen, D. Ø. (2020) · Administrative Sciences · doi:10.3390/admsci10020032 · Cited by 25
Augmented Reality and Television: Dimensions and Themes
Saeghe, P., Abercrombie, G., Weir, B., Clinch, S., Pettifer, S. R., & Stevens, R. (2020) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3391614.3393649 · Cited by 28
“Forward and backward citations were followed in accordance with Wohlin’s guidelines for snowballing [63], resulting in the inclusion of five further publications.”
MethodSerious games for teaching agile methods: A review of multivocal literature
Rodríguez, G., Caino, P. C. G., & Resett, S. A. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22430 · Cited by 32
“Snowballing refers to the use of a paper reference list (backward snowballing) or paper citations to find additional articles (forward snowballing) [74].”
BackgroundA Systematic Literature Review on Blockchain GovernancePreprint
Liu, Y., Lu, Q., Zhu, L., Paik, H.-Y., & Staples, M. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3981282 · Cited by 30
“Hereby, backward and forward snowballing refers to using the references and citations of a paper respectively to find related studies [32].”
MethodHoffman, S. N., Rassaby, M. M., Stein, M. B., & Taylor, C. T. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.01.086 · Cited by 22
Demetracopoulou, V., O’Brien, W. J., Khwaja, N. A., et al. (2022) · Engineering Construction & Architectural Management · doi:10.1108/ecam-05-2022-0455 · Cited by 21
A survey on intelligent management of alerts and incidents in IT services
Yu, Q., Zhao, N., Li, M., Li, Z., Wang, H., Zhang, W., Sui, K., & Pei, D. (2024) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2024.103842 · Cited by 52
Enge, K., Elmquist, E., Caiola, V., Rönnberg, N., Rind, A., Iber, M., Lenzi, S., Lan, F., Höldrich, R., & Aigner, W. (2024) · Computer Graphics Forum · doi:10.1111/cgf.15114 · Cited by 47
“Furthermore, we extended the corpus of relevant papers by snowballing [Woh14], checking all incoming and outgoing references of the articles matching our inclusion criteria.”
MethodData science for job market analysis: A survey on applications and techniques
Rahhal, I., Kassou, I., & Ghogho, M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124101 · Cited by 45
Haslinger, A.-S., Huysveld, S., Cadena, E., & Dewulf, J. P. (2024) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-024-02312-y · Cited by 27
“The decision to include or exclude the article was made after reading the abstract followed by other parts of the paper (Wohlin 2014).”
Explainable biometrics: a systematic literature review
Tucci, C., Greca, A. D., Tortora, G., & Francese, R. (2024) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-024-04856-1 · Cited by 27
Fostering project risk management in SMEs: an emergent framework from a literature review
Testorelli, R., Lima, P. F. D. A., & Verbano, C. (2020) · Production Planning & Control · doi:10.1080/09537287.2020.1859633 · Cited by 29
“…in step 6, using a reference list and citations of the selected papers, according to the guidelines for snowballing in systematic literature studies proposed by Wohlin (Wohlin 2014), and six additional papers were selected accordingly, raising the total number of selected papers to 19 (Table 1).”
MethodDynamic Language Use in Bi/Multilingual Early Childhood Education Contexts
Pontier, R. W., Boruchowski, I. D., & Olivo, L. I. (2020) · Journal of Culture and Values in Education · doi:10.46303/jcve.2020.18 · Cited by 23
“After this initial research, we used the reference sections of relevant articles and dissertations of the 125 original sources to find additional articles that met inclusion criteria but had not surfaced at first (Wohlin, 2014).”
MethodDESIGN FOR LONGEVITY - A FRAMEWORK TO SUPPORT THE DESIGNING OF A PRODUCT'S OPTIMAL LIFETIME
Carlsson, S., Mallalieu, A., Almefelt, L., & Malmqvist, J. (2021) · Proceedings of the Design Society · doi:10.1017/pds.2021.100 · Cited by 29
“The overall goal in this paper is to develop a framework that helps product developers to Design for Longevity , given any durable good (Wikipedia, 2021) within the consumer industry.”
BackgroundAcademic Writing Groups in Higher Education: History and State of Play
Déri, C. E., Tremblay-Wragg, É., & Mathieu-Chartier, S. (2021) · International Journal of Higher Education · doi:10.5430/ijhe.v11n1p85 · Cited by 24
“All of these documents served to find other references drawing from their bibliographies by using a method called “the snowball effect” (Wohlin, 2014), which produced 14 additional articles to analyze.”
Method30 Years of Automated GUI Testing: A Bibliometric Analysis
Rodríguez-Valdés, O., Vos, T. E. J., Aho, P., & Marín, B. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-85347-1_34 · Cited by 27
“Then, a backward snowballing [21] on the 720 papers resulted 50 new papers that were screened based on the title and abstract.”
BackgroundHybrid Metaheuristics for QoS-Aware Service Composition: A Systematic Mapping Study
Naghavipour, H., Tey, K. S., Idris, M. Y. I., Namvar, M., Salleh, R., & Gani, A. (2021) · IEEE Access · doi:10.1109/access.2021.3133505 · Cited by 25
“Moreover, backward and forward [111] done to ensure no imperative study missed.”
BackgroundAdoption of Cloud Computing in E-Government: A Systematic Literature Review
Abied, O., Ibrahim, O., & Kamal, S. N.-I. M. (2022) · Pertanika journal of science & technology · doi:10.47836/pjst.30.1.36 · Cited by 33
“The measures enabled the production of outcomes that underwent snowballing from the explanation given (Wohlin, 2014).”
BackgroundA Framework for Evaluating Dashboards in Healthcare
Zhuang, M., Concannon, D., & Manley, E. (2022) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 21
Evaluation of Systems-of-Systems Software Architectures: State of the Art and Future Perspectives
Santos, D. S., Oliveira, B. R. N., Kazman, R., & Nakagawa, E. Y. (2022) · ACM Computing Surveys · doi:10.1145/3519020 · Cited by 26
“• Third selection: a one-depth backward/forward snowballing [123] was applied to the studies selected in the previous step, and identified another 13 relevant studies.”
Survey on the Past Decade of Technology in Animal Enrichment: A Scoping Review
Kresnye, K. C., Chung, C.-F., Martin, C. F., & Shih, P. C. (2022) · Animals · doi:10.3390/ani12141792 · Cited by 28
“To conduct this review, we utilized snowballing [18] and PRISMA ScR [19].”
MethodConflict management techniques for model merging: a systematic mapping review
Sharbaf, M., Zamani, B., & Sunyé, G. (2022) · Software & Systems Modeling · doi:10.1007/s10270-022-01050-9 · Cited by 35
“To finish the search process, we complemented the results obtained from the last step with a backward and forward snowballing search [44] in order to ensure covering the broad scope of the study.”
MethodNarbaev, T. (2022) · International Journal of Strategic Property Management · doi:10.3846/ijspm.2022.17860 · Cited by 29
“A snowballing search approach refers to using references of an article (going backwards to find prior articles) or citations to this article (going forward to find later articles) to identify additional articles for a literature review study (Wohlin, 2014).”
MethodBIM and real estate valuation: challenges, potentials and lessons for future directions
Jafary, P., Shojaei, D., Rajabifard, A., & Ngo, T. (2022) · Engineering Construction & Architectural Management · doi:10.1108/ecam-07-2022-0642 · Cited by 33
A systematic literature review on Android-specific smells
Wu, Z., Chen, X., & Lee, S. U.-J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111677 · Cited by 27
Comprehensive Review of Methods to Assess Uncertainty in Health Economic Evaluations
Otten, T., Grimm, S. E., Ramaekers, B. L. T., & Joore, M. A. (2023) · PharmacoEconomics · doi:10.1007/s40273-023-01242-1 · Cited by 32
“2 Methods We performed a snowballing review in which we identified articles that may be relevant to the research question by screening references and citations as described by Wohlin [19].”
MethodNadimpalli, V. K., Häuser, F., Bittner, D., Grabinger, L., Staufer, S., & Mottok, J. (2023) · European Conference on Software Engineering Education (ECSEE) · doi:10.1145/3593663.3593681 · Cited by 30
“Snowballing is the collection of additional work based on the references or citation list of a paper [72].”
BackgroundMethodA comparative review on the role of stimuli in idea generation
Blandino, G., Montagna, F., Cantamessa, M., & Colombo, S. (2023) · Artificial intelligence for engineering design analysis and manufacturing · doi:10.1017/s0890060423000124 · Cited by 31
“Methodology for the review The Snowballing methodology (Wohlin, 2014) adopted for initially selecting contributions highlighted the rapid diffusion that occurred since 2016 of the use of methods from neurophysiology in the engineering and industrial design literature.”
Caruana, A., Bandara, M., Musiał, K., Catchpoole, D. R., & Kennedy, P. J. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102642 · Cited by 35
“Once the evidence was finalised, one author extracted results from the identified literature and a second author reviewed and verified the results to ensure accuracy.”
MethodFounders and the success of start-ups: An integrative review
Aryadita, H., Sukoco, B. M., & Lyver, M. J. (2023) · Cogent Business & Management · doi:10.1080/23311975.2023.2284451 · Cited by 35
“As a result, to get a broader idea of the factors that influence the success of start-ups, a backward snowball strategy was used to select papers related to the topic and research objectives (Wohlin, 2014).”
MethodA scoping review of professional skills development in engineering education from 1980–2020
Chadha, D., & Heng, J. Y. Y. (2024) · Cogent Education · doi:10.1080/2331186x.2024.2309738 · Cited by 34
“In this way, both forward snow-balling (identifying individuals who cited work) and backwards snowballing (identifying earlier work from reference lists) were applied (Wohlin, 2014).”
Smart Homes App Vulnerabilities, Threats, and Solutions: A Systematic Literature Review
Ansari, A. M., Nazir, M., & Mustafa, K. (2024) · Journal of Network and Systems Management · doi:10.1007/s10922-024-09803-1 · Cited by 29
Waseem, M., Ahmad, A., Liang, P., Akbar, M. A., Khan, A. A., Ahmad, I., Setälä, M., & Mikkonen, T. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112558 · Cited by 36
(Sprayed) concrete production in life cycle assessments: a systematic literature review
Saade, M. R. M., Passer, A., & Mittermayr, F. (2019) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-019-01676-w · Cited by 23
“(Wohlin 2014), we checked for relevant papers that were”
MethodA Systematic Mapping of Translation-Enabling Technologies for Sign Languages
Naranjo-Zeledón, L., Peral, J., Ferrández, A., & Chacón-Rivas, M. (2019) · Electronics · doi:10.3390/electronics8091047 · Cited by 27
“We have opted for the first option since snowballing is most used to extend a review already carried out, and it still has some important features left for further research, like that of identifying a good start set [71].”
BackgroundSense of School Belonging and Paediatric Illness: A Scoping Review
Tomberli, L., & Ciucci, E. (2021) · Continuity in Education · doi:10.5334/cie.32 · Cited by 21
Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study
Gemeinhardt, F., Garmendia, A., Wimmer, M., Weder, B., & Leymann, F. (2024) · ACM Computing Surveys · doi:10.1145/3620668 · Cited by 22
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
MethodLearning software configuration spaces: A systematic literature review
Pereira, J. A., Acher, M., Martin, H., Jezéquél, J., Botterweck, G., & Ventresque, A. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111044 · Cited by 179
“Thus, apart from the automatic search, we also consider the use of snowballing [111] as a complementary approach.”
BackgroundIdentity in Higher Computer Education Research: A Systematic Literature Review
Große-Bölting, G., Gerstenberger, D., Gildehaus, L., Mühling, A., & Schulte, C. (2023) · ACM Transactions on Computing Education · doi:10.1145/3606707 · Cited by 26
“Based on the seed corpus formed by the database search, backward and forward snowballing [131] were performed.”
BackgroundMethodBehind the Intent of Extract Method Refactoring: A Systematic Literature Review
AlOmar, E. A., Mkaouer, M. W., & Ouni, A. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3345800 · Cited by 31
“Based on established guidelines [35], [36], [50], [51], [52], we performed Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
MethodInfluentialLiu, J., Duan, L., Lin, S., Ji, M., & Zhao, J. (2024) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-024-10140-4 · Cited by 25
Lee, C., Kim, H. F., & Lee, B. G. (2024) · Electronics · doi:10.3390/electronics13142885 · Cited by 61
Akhdar, A. E., Baidada, C., Kartit, A., Hanine, M., García, C., Lara, R. G., & Ashraf, I. (2024) · Sensors · doi:10.3390/s24206771 · Cited by 41
“Snowballing, which involves reviewing references of identified studies [18], is useful for finding related research but can be time-consuming and may not capture all relevant studies.”
BackgroundSurvey and Benchmark of Anomaly Detection in Business Processes
Guan, W., Cao, J., Zhao, H., Gu, Y., & Qian, S. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3484159 · Cited by 24
“As various authors may employ diverse terminologies that are not covered by our search terms, we apply backward snow-balling [25], including 13 studies that were cited by previously published papers following the ECs.”
Lee, H., Huang, Y., & Wu, T. (2025) · European Journal of Education · doi:10.1111/ejed.70262 · Cited by 41
Larooij, M., & Törnberg, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11412-6 · Cited by 90
“To expand the search and ensure comprehensive coverage, we employed backward snowballing (Wohlin 2014), reviewing the reference lists of the included papers, after having filtered the initial papers based on the procedure below.”
Spennemann, D. H. R. (2019) · Flora · doi:10.1016/j.flora.2019.151468 · Cited by 23
Reis, D. A., Fleury, A. L., & de Carvalho, M. M. (2019) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2019.2917406 · Cited by 27
“sampling step, snowballing was applied for identifying the key references cited in the initial sample [118], resulting not only in articles, but also in core books, that were presented in the”
MethodData Digitalisation in the Open-Pit Mining Industry: A Scoping Review
Duarte, J., Rodrigues, F., & Baptista, J. S. (2020) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-020-09493-3 · Cited by 20
“After the definition of the main exclusion and inclusion criteria, all of the references of the selected articles were checked in order to find more relevant information, according to the snowballing technique procedure [23].”
MethodAnalyses of energy-GDP-export nexus: The way-forward
Shakeel, M. (2020) · Energy · doi:10.1016/j.energy.2020.119280 · Cited by 23
Conversational Agents for Elderly Interaction
Pinto, N., França, J. B. D. S., Sousa, H. P., Vivacqua, A. S., & García, A. C. B. (2021) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd49262.2021.9437883 · Cited by 29
“Then, we used the backward snowballing method based on [16] and added more 17 papers.”
MethodMoroni, F., Baldetti, L., Kabali, C., Briguori, C., Maioli, M., Toso, A., Brilakis, E. S., Gurm, H. S., Bagur, R., & Azzalini, L. (2021) · Journal of the American Heart Association · doi:10.1161/jaha.121.021342 · Cited by 21
Migration from Monolith to Microservices: Benchmarking a Case Study.
Bjørndal, N., Mazzara, M., Bucchiarone, A., Dragoni, N., & Dustdar, S. (2021) · The Journal of Object Technology · doi:10.5381/jot.2021.20.2.a3 · Cited by 25
Metaphoraction: Support Gesture-based Interaction Design with Metaphorical Meanings
Sun, Z., Wang, S., Liu, C., & Ma, X. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3511892 · Cited by 27
“After getting an initial set of papers, we obtain 13 additional papers relevant to the target topic by following the snowballing method in systematic reviews [151].”
Method
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.