Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 3,101–3,200 of 3,934
A Systematic Mapping Study of Innovative Cloud Applications
Odun-Ayo, I. A., Williams, T. M., Iheanetu, O. U., Odusami, M., & Bogle, S. (2020) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/811/1/012034 · Cited by 1
“In building the classification scheme, a thematic analysis approach, which aimed at identifying, analyzing and reporting themes within the studies under consideration was adopted [19].”
MethodDefining Security Metrics To Evaluate Electronic Health Records Systems: A Case Study in Chile
Márquez, G., Taramasco, C., & Hernán Astudillo, R. (2020) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c50368.2020.00038 · Cited by 1
“We executed the snowballing process in academic literature [13] to gather more studies.”
MethodDesign for Societal Resilience: The Risk Evaluation Diversity-Aiding Approach (RED-A)
Onencan, A. M., Liu, L. E., & de Walle, B. V. (2020) · Sustainability · doi:10.3390/su12135461 · Cited by 2
“Thereafter, we used forward snowballing to check for articles that cited publications that had passed the initial assessment [46].”
MethodAcosta, P., & Maroto, J. L. S. F. (2020) · Cited by 1
Duarte, J., Branco, J. C., & Baptista, J. S. (2020) · International Journal of Occupational and Environmental Safety · doi:10.24840/2184-0954_004.002_0007 · Cited by 1
“…characteristics The study will be conducted in two phases: 1) only English-written literature published between January 2010 and January 2021 will be searched for in peer-reviewed journals, 2) The snowballing technique (Wohlin & Claes, 2014) will be used to search for any other relevant studies.”
MethodIntegrating Complete Locomotive Assistance and IoT-Based Health Care for the Disabled
Aravinthkumar, S., Chandel, A. S., & Al-Absi, A. A. (2020) · Lecture notes in networks and systems · doi:10.1007/978-981-15-7990-5_35 · Cited by 1
Big data y ciencia de diseño: oportunidades de investigación
Zeledón, L. N. (2020) · Cited by 1
Revisiting the Bad Smell and Refactoring Relationship: A Systematic Literature Review.
Tavares, C. S., Santana, A., Figueiredo, E., & Bigonha, M. A. S. (2020) · Conferencia Iberoamericana de Software Engineering · Cited by 5
Mendes, Y., de Oliveira, D., & Ströele, V. (2021) · Journal of Information and Data Management · doi:10.5753/jidm.2020.2017 · Cited by 1
“To mitigate threats to this study regarding papers that share the same scope as this SLM, but were either not in any of the knowledge bases used in this research, or not captured by the search string, we conducted one iteration of backward and forward snowballing [Wohlin 2014].”
MethodWHAT ARE THE BARRIERS FOR TEACHING COMPUTATIONAL THINKING?
Muilu, M., Mehtälä, S., Pawlowski, J. Μ., Dimitrakopoulou, D., Pilv, M., Clements, K., & Idzik, M. (2021) · INTED proceedings · doi:10.21125/inted.2021.1184 · Cited by 1
Find the Way in the Jungle of Quality of Service in Industrial Cloud: A Systematic Mapping Study
Latifaj, M., Ciccozzi, F., & Sentilles, S. (2021) · International Conference on Cloud Computing and Services Science · doi:10.5220/0010380401510160 · Cited by 3
“Charters (2007), and Wohlin et al. (2012). Furthermore, we defined a classification framework for the”
MethodInfluentialBacksourcing of Software Development -- A Systematic Literature Review
Molléri, J. S., Lassenius, C., & Jørgensen, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11564 · Cited by 2
“To address this problem, we used backward and forward snowballing [36] to search for additional studies.”
Method(Inter) profissionais conectados para a aprendizagem colaborativa online: Uma revisão de escopo
Mota, D. P. M., & Rocha, D. G. (2021) · Research Society and Development · doi:10.33448/rsd-v10i8.17610
Competências necessárias para suportar a Indústria 4.0
de Barros, M. I., Cassel, R. A., & da Silva, B. P. (2021) · Anais ... Encontro Nacional de Engenharia de Produção/Anais do Encontro Nacional de Engenharia de Produção · doi:10.14488/enegep2021_tn_stp_363_1873_41669 · Cited by 1
“Esta revisão sistemática adotou o sistema de bola de neve (WOHLIN, 2014), isso significa que quando os artigos relacionados não estavam na pesquisa original, mas foram referenciados dentro os artigos identificados, foram posteriormente adicionados.”
BackgroundOpportunity Space for Work-related Crime from Procurement to Production
Selvik, A. S., Edvardsen, M. M., Lædre, O., et al. (2022) · Procedia Computer Science · doi:10.1016/j.procs.2021.12.090 · Cited by 1
Steglich, C., dos Santos, R. P., Marczak, S. D. S., et al. (2021) · iSys - Brazilian Journal of Information Systems · doi:10.5753/isys.2021.2015 · Cited by 1
Applying guidelines for system modelling in Event-B - a systematic literature review
Olszewska, M., Manninen, F. S., Edmunds, A., et al. (2020) · International Journal of Critical Computer-Based Systems · doi:10.1504/ijccbs.2020.108668 · Cited by 1
Software Architecture for Smart Cities: A Systematic Literature Review of Quality Attributes
Abdullahi, M. S. (2022) · SLU Journal of Science and Technology · doi:10.56471/slujst.v4i.275 · Cited by 2
Capability Evaluation in Context Agnostic Agile Assessment
Tuncel, D., Körner, C., & Plösch, R. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-15559-8_34 · Cited by 1
TYPES OF PARTLY CONFIGURABLE PRODUCTS IN HIGH-VARIETY, LOW-VOLUME CONTEXT
Heikkinen, T., & Juuti, T. S. (2023) · Proceedings of the Design Society · doi:10.1017/pds.2023.367 · Cited by 1
“Additionally, backward snowballing was utilized to capture relevant referenced information (Wohlin, 2014) regarding the research questions and the objective.”
Methoddos Santos, I. N. G., & de Andrade, M. D. (2023) · Scientia Iuris · doi:10.5433/2178-8189.2023v27n3p94-118 · Cited by 1
Quality Attribute Tradeoff in Learning Infrastructure Scaling
van Schuppen, C. (2015) · Utrecht University Repository (Utrecht University) · Cited by 1
Microplastics in the Oresund Region of the Baltic Sea - sources and effects
Bystedt, S. (2018) · Lund University Publications Student Papers (Lund University) · Cited by 2
Enhancing Trust –A Unified Meta-Model for Software Security Vulnerability Analysis
Alqahtani, S. S. (2018) · Spectrum Research Repository (Concordia University) · Cited by 1
Tool Support of Formal Methods for Privacy by DesignPreprint
Schupp, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.11092 · Cited by 1
“…approach consists of two stages, where stage 1 conducts a systematic search to obtain an initial list ([56, 57]) and stage 2, called snowballing ([104]), traces the forward and backward references to and from Figure 1: Selection process: stage 1 (search), stage 2 (forward and backward…”
BackgroundTowards a Strategy for Supporting the Engineering of Contemporary Software SystemsPreprint
Motta, R. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11741 · Cited by 1
“It refers to using the reference list of an article or its citations to identify additional material (Wohlin, 2014).”
BackgroundInfluentialA survey on software testabilityPreprint
Garousi, V., Felderer, M., & Kılıçaslan, F. N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02201 · Cited by 1
Designing Security and Privacy Requirements in Internet of Things: A SurveyPreprint
Alhirabi, N., RANA, O. F., & Perera, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09911 · Cited by 3
Serious games como apoio para avaliação e treinamento cognitivo na senescência
do Amaral, L. A. (2020) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2020.tde-08062020-092328 · Cited by 3
Madsen, D. (2020) · Social Science Research Network · doi:10.2139/ssrn.3738866 · Cited by 1
Software Engineering for Robotic Systems:a systematic mapping studyPreprint
dos Santos, M. G., & Petrillo, F. D. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12520 · Cited by 4
“A forward snowballing refers to identifying new papers based on papers that cite citing the paper examined, and backward snowballing is the process to evaluate the references for the paper in the set [12].”
Privacy-Aware Internet of Things Notices in Shared Spaces: A SurveyPreprint
Muhander, B. A., Wiese, J., RANA, O. F., & Perera, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13633 · Cited by 4
Vyshedsky, C. (2021) · Cited by 1
4th Symposium on Occupational Safety and Health Proceedings Book
Pinho, O., Baptista, J. S., Bustos, D., Niquice, F., Branco, J. C., Duarte, J., Fernandes, M., Martins, R., Vasconcelos, S., Teixeira, T., et al. (2021) · doi:10.24840/978-972-752-279-8 · Cited by 1
Blockchain Driven Access Control Mechanisms, Models and Frameworks: A State of the Art ReviewPreprint
Bashir, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3704466
The Impact of Traceability on Software Maintenance and Evolution: A Mapping StudyPreprint
Tian, F., Wang, T., Liang, P., Wang, C., Khan, A. A., & Babar, M. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02133 · Cited by 1
Timely evaluation in international development
Vigneri, M. (2022) · doi:10.51744/cmwp7 · Cited by 1
A systematic review of risk factors of type 2 diabetes mellitus in the Ashanti region of GhanaPreprint
Katey, D. (2022) · Research Square · doi:10.21203/rs.3.rs-1611015/v1 · Cited by 1
Green, J. (2021) · doi:10.36227/techrxiv.15138444 · Cited by 1
“The second step was a ”snowball” search [9] using the reference section of the selected papers.”
MethodGupta, V. (2022) · UPM Digital Archive (Technical University of Madrid) · doi:10.20868/upm.thesis.70481 · Cited by 5
Ash, A. K., Zhao, Y., Piña‐Covarrubias, E., Doncaster, C. P., Spake, R., Gill, R., & Nichols, C. (2022) · doi:10.32942/osf.io/5qrgv · Cited by 1
Sound Zone Interaction in Homes: Studying and Designing for Modification of Personal Soundscapes
Johansen, S. S. (2021) · VBN Forskningsportal (Aalborg Universitet) · doi:10.54337/aau455013535 · Cited by 3
Seth, M., Jalo, H., Högstedt, Å., et al. (2022) · doi:10.2196/preprints.40243 · Cited by 1
Hawes, D. (2022) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2022-15050 · Cited by 2
Eramo, R., Tucci, M., Pompeo, D. D., et al. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4088422 · Cited by 1
Holistic Perspectives on Safety of Automated Driving Systems - Methods for Provision of EvidencePreprint
Gyllenhammar, M., de Campos, G. R., & Törngren, M. (2022) · doi:10.36227/techrxiv.20331243.v1 · Cited by 1
Holistic Perspectives on Safety of Automated Driving Systems - Methods for Provision of EvidencePreprint
Gyllenhammar, M., de Campos, G. R., & Törngren, M. (2022) · doi:10.36227/techrxiv.20331243 · Cited by 4
A novel Road User Safety Field Theory for traffic safety assessment applying video analytics
Arun, A. (2022) · Queensland University of Technology · doi:10.5204/thesis.eprints.234039 · Cited by 1
The Leadership Practices of Nurses in the New Zealand Hospital Ward: A Focused Ethnography
Lindsay, N. (2023) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.22827629 · Cited by 3
Ethical reasoning methods for ICT: the technical reportPreprint
España, S., van der Maaten, C., Gulden, J., & Pastor, Ó. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ebmtp · Cited by 2
Yang, L. (2021) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.5821/dissertation-2117-355671 · Cited by 2
de Wit, M. M., Martins, A. D., Delecroix, C., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.26.564175 · Cited by 1
A Systematic Mapping Study on Automated Negotiation for Autonomous Intelligent AgentsPreprint
Memon, M. A., Scoccia, G. L., & Autili, M. (2023) · Research Square · doi:10.21203/rs.3.rs-3442930/v1 · Cited by 1
Technical Debt Management Automation: State of the Art and Future PerspectivesPreprint
Biazotto, J. P., Feitosa, D., Avgeriou, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.18449 · Cited by 1
The contribution of community-based programmes to health
Pardoel, Z. (2024) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.925136990 · Cited by 1
The Illusion of the Shared Electric Automated Vehicles TransitionPreprint
Milakis, D., & Seibert, D. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4730663 · Cited by 1
Stamatiadou, V., Mazaris, A. D., & Katsanevakis, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4782372 · Cited by 1
Characterising Reproducibility Debt in Scientific Software:A Systematic Literature ReviewPreprint
Hassan, Z., Treude, C., Norrish, M., Williams, G., & Potanin, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4801433 · Cited by 1
Software Architecture-Based Self-Adaptation in RoboticsPreprint
Alberts, E., Gerostathopoulos, I., Malavolta, I., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4805883 · Cited by 1
Iliadis, E. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4882101 · Cited by 1
Cinquini, L., Pigatto, G., Tenucci, A., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5024161 · Cited by 1
Masood, R., Dev, V., Gee, M., et al. (2025) · medRxiv · doi:10.1101/2025.03.22.25324427 · Cited by 1
Methodology for Systematic Reviews in EducationPreprint
Huang, R., & Yin, Y. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/az5xm_v1 · Cited by 2
Improving the resilience of the constrained Internet of Things : a moving target defense approach
Navas, R. E. (2020) · theses.fr (ABES) · doi:10.70675/ec975c75z1b33z4693zb02cz4012da71b29f · Cited by 3
Study of Qualimetry essentials applied to embedded software development
Argotti, Y. (2021) · theses.fr (ABES) · doi:10.70675/5b4ce984z5056z4d55za061zda2883129b4d · Cited by 2
SCORE: A Framework for Quantifying Diegesis in Situated Visualization for Augmented Reality
Hasan, T., Hasan, K., & Ens, B. M. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791911 · Cited by 1
“We utilized a snowballing-based literature review methodology [15, 77, 84] to conduct a structured search, a standard approach in HCI for constructing design spaces, employing both backward and forward citation chasing.”
A Scoping Survey on Augmented Display Systems
Raza, M., Ketsoi, V., & Reilly, D. F. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790680 · Cited by 1
Node.js in Open Source projects on Github : A literature study and exploratory case study
Åkesson, A., & Lewenhagen, K. (2015) · DiVA (Blekinge Institute of Technology) · Cited by 1
To Ban or Not to Ban? How Open Source Projects Govern GenAI Contributions
Yang, W., He, R., & Zhou, M. (2026) · Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3832783.3837448 · Cited by 1
On the facets of stakeholder inertia: A literature review
Hoffman, K., Wnuk, K., & Callele, D. (2014) · doi:10.1109/iwspm.2014.6891065
“As a last step, we explored the references of the directly related articles using the snowballing method [34] – one iteration of reading the references identified an additional 2 related articles and 12 promising articles(1).”
MethodÅ investere i volatilitet : egenskaper, marked og nytte i porteføljesammenheng
Aass, F. B., & Døvik, H. (2016) · NORA - Norwegian Open Research Archives
Pinciroli, F., Barros‐Justo, J. L., Zeligueta, L., & Pma, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02653 · Cited by 4
“[14] D. Badampudi, C. Wohlin, and K. Petersen, “Experiences from using snowballing and database searches in systematic literature studies,” Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, pp. 1–10, 2015.”
MethodInfluentialBarros‐Justo, J. L., Sepúlveda, S., Martínez-Araujo, N., & González-García, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00822 · Cited by 1
“The snowballing search will be conducted according to the guidelines by [10], and will consist of the following steps:”
MethodThe role of governance mechanisms on the diffusion of innovation in healthcare networks
Okeke, U. (2017) · UWE Research Repository (UWE Bristol) · Cited by 1
Supporting architectural design of acknowledged SoS
Gonçalves, M. B. (2017) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2017.tde-11012017-162252
Giraldo, F. D. (2017) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/90628
“This analysis was performed through a structured literature review using backward snowballing (Wohlin, 2014) on scientific publications and grey literature (non-scientific publications).”
MethodScalability Drivers in Requirements Engineering
Chaitanya, K. K. (2016) · DiVA (Blekinge Institute of Technology)
Sánchez, C. F. (2017) · UPM Digital Archive (Technical University of Madrid) · doi:10.20868/upm.thesis.49390
Towards a practice-based understanding of organizational memory
Kravčenko, D. (2026) · Warwick Research Archive Portal (University of Warwick) · doi:10.82444/warw.33751180
van Ommering, C. J., de Klerk, S., & Veldkamp, B. P. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-25264-9_13
“An overview of the articles and handbooks, that were included in the current study, is provided in table 4. by Wohlin (2014) was used to find more relevant literature on the topic of test security.”
MethodO'Shea, G. (2018) · Staff Publications Online (The Tavistock and Portman) · Cited by 6
Silva, L., Lima, J., Cacho, N., et al. (2018) · doi:10.5753/wbci.2018.3231
“Terceiro, não foi realizado um processo de snowballing, uma técnica que consiste na verificação das listas de referências de cada um dos estudos selecionados a fim de encontrar estudos adicionais que não foram recuperados pela busca automática [Wohlin 2014].”
BackgroundBarros‐Justo, J. L., Rincón, L., Villota, Á., & Assunção, W. K. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01861
“3 Snowballing The first author will perform a forward and backward snowballing search process as described in [15].”
MethodCvetkovic, M., Duarte, J., Soares, D. P., et al. (2019) · International Journal of Occupational and Environmental Safety · doi:10.24840/2184-0954_003.001_0008
“The process will be repeated until no more relevant results are obtained (Wohlin, 2014).”
BackgroundNogueira, T. D. C., Duarte, L., Los, S., Luz, J., & Ferreira, D. J. (2018) · doi:10.5753/washes.2018.3470
“Dessa forma, para responder `a quest˜ao de pesquisa Q01, esta revis˜ao sistem´atica da literatura utilizou-se a abordagem proposta por [Wohlin 2014].”
BackgroundMooney, J. (2019) · Arrow@dit (Dublin Institute of Technology) · doi:10.13025/17872
Is Fragmentation a Threat to the Success of the Internet of Things?Preprint
Aly, M., Khomh, F., Guéhéneuc, Y., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07355
“3) Round 3: Following existing guidelines [50], [52], we perform a snowball search using the reference list of our studies and citations obtained from the previous Round 2 to identify new studies and decide whether to include additional paper(s) or article(s); such technique helps us not to miss important and relevant…”
MethodA Review on Software Architectures for Heterogeneous PlatformsPreprint
Andrade, H., & Crnković, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.01695 · Cited by 1
“In order to minimize the threat of missing relevant studies in the field, we also conducted the snowballing procedure [7] and obtained an additional 16 studies, resulting in a total of 116 studies.”
ResultThe impact of Use Cases in real-world software development projects: A systematic mapping studyPreprint
Barros‐Justo, J. L., Benitti, F. B. V., & Tiwari, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06754
“We conducted a snowballing search using the papers selected from the automated search strategy as the initial set of seeds, following the guidelines proposed in [47].”
MethodScatalon, L. P. (2019) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2019.tde-29082019-095834
A Systematic Literature Review of Test Breakage Prevention and Repair Techniques
Imtiaz, J., Sherin, S., Khan, M. U., & Iqbal, M. Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10750
Context-Augmented Software Development Projects: Literature Review and Preliminary FrameworkPreprint
Melo, G., Alencar, P. S. C., & Cowan, D. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08167
“Therefore, we also implemented snowballing [18], as an attempt to mitigate this problem with the term context.”
MethodSilva, L., & Néris, V. P. D. A. (2019) · doi:10.5753/ihc.2019.8412
“O procedimento de MS Snowballing, ou em português “bola de neve”, é realizado considerando a lista de referências e citações de um artigo, inicialmente selecionado, para encontrar outros artigos que se relacionam com ele [30].”
BackgroundSemantic-aware EWC Code Recommender System for Industrial Symbiosis Marketplace
Prakoso, D. W. (2019)
Métricas de Tamanho de Software com Métodos Ágeis no Setor Publico: uma Revisão Sistemática
Mendonça, A., & Alencar, F. (2019) · Workshop em Engenharia de Requisitos (WER) · doi:10.29327/1298731.22-14
“Neste ponto do processo de pesquisa, aplicamos a técnica snowballing [21] para melhorar nossos resultados.”
BackgroundMeasurement of Interpersonal Trust in Global Software Development: SLR ProtocolPreprint
Zapata, S., Barros‐Justo, J. L., Maturro, G., & Sepúlveda, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04974
Paines, P. D. A., Vignochi, L., & Ceballos, M. H. (2020) · Editora Poisson eBooks · doi:10.36229/978-85-7042-210-1.cap.08
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.