Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 3,301–3,400 of 3,934
The Asian Conference on Arts & Humanities 2021 Official Conference Proceedings
(2021) · The Asian Conference on the Arts, Humanities and Social Sciences Conference proceedings/Asian Conference on the Social Sciences official conference proceedings/Asian Conference on Arts & Humanities official conference proceedings · doi:10.22492/issn.2186-229x.2021 · Cited by 1
(2019) · PeerJ Computer Science · doi:10.7287/peerj-cs.230v0.1/reviews/2
“We employed a snowball 121 method (Wohlin, 2014) where we started with some initial set and used their references, both 122 backward and forward iterations, until we no longer found significant new work.”
Assessing Social Behaviour Towards Near-Body Product Users in the Wild: A Review of Methods
Boeck, M. D., Vleugels, J., Rooy, D. V., & Vaes, K. (2022) · Proceedings of the Design Society · doi:10.1017/pds.2022.212
“Because the search for research methods was very wide ranging, we used both a forward snowball search strategy to see where a relevant article was cited, and a backward snowball strategy to see which citations were used in that article (Wohlin, 2014).”
MethodDigital Discrimination in the Sharing Economy: Evidence, Policy, and Feature Analysis
Tushev, M. (2022) · Louisiana State University · doi:10.31390/gradschool_dissertations.5760
Salgado, A. D. L. (2022) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2022.tde-02062022-142408 · Cited by 3
Kaplan, A. (2022) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3530019.3535345 · Cited by 1
Narrative Visualizations Best Practices and Evaluation: A Systematic Mapping StudyPreprint
Airaldi, A. L., Irrazábal, E., & Diaz-Pace, J. A. (2022) · Research Square · doi:10.21203/rs.3.rs-1735564/v1 · Cited by 2
Rethinking the Design of Online Professor Reputation Systems
Tatum, H. (2022) · Louisiana State University · doi:10.31390/gradschool_theses.5515
“To reduce the risk of false negatives (paper omission), a backward-snowballing was applied on the set of included primary studies [72].”
MethodGame-based Learning and Co-Design with and for Newcomer Children’s Social Adjustment
Bani-Taha, O. (2022) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2022-15078
Peran Sistem Informasi untuk Mendukung Pengembangan Karir dan Kompetensi (Literature Review)
Susanto, L. J., Pujianto, P., & Asmanto, B. (2022) · Jurnal Ilmiah Sistem Informasi (JISI) · doi:10.24127/jisi.v1i1.1954
User Interface Modelling Languages for Normalised Systems: Systematic Literature Review
Slifka, J., & Pergl, R. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-04829-6_31
REALÓGICA: A SERIOUS GAME TO TEACH PROPOSITIONAL LOGIC FOR ELEMENTARY STUDENTS
SILVA, J. A. R. R. D., & da Silva, P. C. (2022) · CONTECSI - International Conference on Information Systems and Technology Management · doi:10.5748/18contecsi/pse/soc/6718
Sánchez-Planelles, J. (2022) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/181474
Understanding Black-Box Attacks Against Object Detectors from a User’s Perspective
Midtlid, K. A., Åsheim, J., & Li, J. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-14179-9_18
Environmental and Occupational Safety and Hygiene KPI in the Mining Industry—A Short Review
Duarte, J., Branco, J. C., Rodrigues, F., et al. (2022) · Studies in systems, decision and control · doi:10.1007/978-3-031-12547-8_41
Improving the Source-to-Sea approach for marine litter in Brazil
Alencar, M. V. (2022) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.21.2022.tde-19092022-144406 · Cited by 1
Guerra, A. A. C. (2021) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.12.2021.tde-07102022-192233
Adaptive user interfaces in systems targeting chronic disease: a systematic literature review
Wang, W., Khalajzadeh, H., Madugalla, A. K., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09340
Mapping research approaches to data practices in schools
Krein, U., Hartenstein, A., & Schiefner-Rohs, M. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.1023096 · Cited by 1
“To fill remaining gaps in the literature review, additional searches of the reference lists of the selected datasets were conducted using the snowballing method (Wohlin, 2014).”
MethodVisual Politics: Visual Political Communication on Social Media
Farkas, X. (2022) · CUB Ph.D. Dissertations (Corvinus University of Budapest) · doi:10.14267/phd.2022065
Santos, I. D., Amaral, D. C., da Silva, S. L., & Foureaux, E. (2022) · Revista Brasileira de Computação Aplicada · doi:10.5335/rbca.v14i3.13501
Linux kernel device driver testing
Schmitt, M. (2022) · doi:10.11606/d.45.2022.tde-30112022-152524
Counteracting Dominant Design Through Intersectional Feminist Thought
Chopra, M. (2022) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2022-15301 · Cited by 1
BUTLER, C. (2022) · Cronfa (Swansea University) · doi:10.23889/suthesis.62146
Discalculia e Tecnologias Digitais Educacionais: uma Revisão Sistemática de Literatura
Neto, J. C., Felizardo, K. R., & Blanco, M. B. (2022) · Informática na educação teoria & prática · doi:10.22456/1982-1654.119335
“Técnicas complementares, como snowballing (WOHLIN, 2014), também podem ser adotadas; 2.2.”
BackgroundInfluentialAlbuquerque, D. W., Chagas, F., Guimarães, E. T., Tonin, G., Perkusich, M., de Almeida, H. O., & Perkusich, Â. (2022) · doi:10.5753/ise.2022.227051
“While this strategy significantly increased the search effort [25], it enabled us to find a comprehensive set of relevant studies.”
BackgroundGupta, V. (2022) · Synthesis lectures on technology, management, and entrepreneurship · doi:10.1007/978-3-031-18322-5_3
Norambuena, V. H. (2022) · UPM Digital Archive (Technical University of Madrid) · doi:10.20868/upm.thesis.72358 · Cited by 1
Visual Analytics for Explainable and Trustworthy Machine Learning
Chatzimparmpas, A. (2023) · doi:10.15626/lud.482.2023
de Lima, L. F., & Peres, L. M. (2022) · SBC Reviews on Computer Science · doi:10.5753/reviews.2022.2724
Dowlut, N., & Gobin-Rahimbux, B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4350291
Schöberl, M., Tuyisabe, S., Hollauer, C., et al. (2022) · 2022 IEEE 28th International Conference on Engineering, Technology and Innovation (ICE/ITMC) & 31st International Association For Management of Technology (IAMOT) Joint Conference · doi:10.1109/ice/itmc-iamot55089.2022.10033150
“A snowballing technique was applied; thus, the references of the first relevant papers were used to identify additional relevant papers [25].”
MethodStrategy for Implementation of the Security Maturity Model in e-Government Systems in Indonesia
Nastiti, T. I. (2022) · InComTech Jurnal Telekomunikasi dan Komputer · doi:10.22441/incomtech.v12i3.15214
Kozloski, C. L., Vaghetti, M. A. O., & Torres, M. C. A. (2022) · Encontro Nacional de Tecnologia do Ambiente Construído · doi:10.46421/entac.v19i1.2223
Abidin, A. N. Z., Aris, M. S. B. M., Razali, A., & Ismail, N. B. (2022) · Malaysian Journal of Medicine and Health Sciences · doi:10.47836/mjmhs.18.s15.43
Härer, F., & Herzwurm, G. (2023) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-023-00966-0 · Cited by 1
“Die dargestellten Probleme wurden mittels zentraler Schlüsselliteratur im Untersuchungsumfeld nach der Taxonomie von Cooper (1988) und der zusätzlichen Literatursystematik „Snowballing“ nach Wohlin (2014) identifiziert.”
BackgroundA Review of Technology-Supported Classroom Observation in Teaching Evaluation
Zhang, Q., Liao, J., Liu, G., & Ke, Y. (2022) · International Conference of Educational Innovation through Technology (EITT) · doi:10.1109/eitt57407.2022.00029 · Cited by 2
“Snowballing method in literature review is a technique for identifying additional papers based on the reference list of the current paper and the citations to the paper [7].”
MethodTechnical Interviews: Another Barrier to Broadening Participation in Computing?
Lunn, S. J. (2021) · FIU - Digital Commons (Florida International University) · doi:10.25148/etd.fidc010298 · Cited by 1
Towards an Integrative Assessment Model for Port Sustainability Decisions: A Systematic Review
Wu, X., Liu, S., Hong, S., & Chen, H. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-32534-2_16
Dillarstone, H., Brown, L. J., & Flores, E. C. (2023) · doi:10.31223/x54m30
Jilina, P. (2023) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.23280740 · Cited by 1
Five Agile Methods in Software Development: A Review Paper
Abdala, T. M., & Barka, A. A. (2017) · مجلة الجامعة الأسمرية · doi:10.59743/aujas.v2i1.1056
A Review of COVID-19 Data Usage in Correctional Institutions From Developing Countries
Nunes, L. D. S., Moreno, E. D., & Carneiro, G. D. F. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3280912
“The snowballing technique [50] must meet one of the two criteria as follows.”
MethodEnhancement of Energy Efficiency Using Ai Techniques: Systematic Literature ReviewPreprint
Bashir, F., Aleem, S., & Khan, R. Q. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4473009
Polzer, T., Nolte, I. M., & Seiwald, J. (2023) · Revue Internationale des Sciences Administratives · doi:10.3917/risa.892.0147
Uma Abordagem para Alinhamento de Requisitos de Segurança e Proteção de Sistemas IoT Críticos
Veiga, E. F. (2023) · Congresso Ibero-Americano em Engenharia de Software (CIbSE) · doi:10.5753/cibse.2023.24712 · Cited by 3
Digitalización del Sector de la Consultoría: impulsores y facilitadores internos y externos
Brazo, P., Velicia-Martín, F., & Palos‐Sánchez, P. R. (2023) · Estudios Gerenciales · doi:10.18046/j.estger.2023.167.5498 · Cited by 2
Características de las Bases de Datos para el Cómputo en la Nube: Un Estudio Secundario
Facultad de Matemáticas, U. A. D. Y., Vera, R. A. A., Sánchez, N. G., et al. (2023) · RISTI - Revista Ibérica de Sistemas e Tecnologias de Informação · doi:10.17013/risti.49.115-130
“Para incrementar y mejorar el conjunto de estudios pertinentes para la RSL, se decidió por el grupo de investigadores, utilizar la técnica denominada bola de nieve hacia adelante (Wohlin, 2014).”
BackgroundHuang, S.-J., & Lin, C.-T. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4511875
ПОДХОДЫ К ПРЕПОДАВАНИЮ ФАРМАКОНАДЗОРА В ВЫСШИХ УЧЕБНЫХ ЗАВЕДЕНИЯХ
Матвеева, Е. В., Ткаченко, Е. В., Дьяченко, А. И., et al. (2023) · Farmaciâ Kazahstana · doi:10.53511/pharmkaz.2023.60.36.032
Vulnerability Detection for Smart Contracts: A Systematic Literature ReviewPreprint
Vidal, F. R., Ivaki, N., & Laranjeiro, N. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4511778
Flores-Calero, M. J., Carrillo, J. L., Espinel, P., Arcos, A., Arias-Almeida, I., Cabezas, C. F., & Corral, A. (2023) · Journal of Applied Research and Technology · doi:10.22201/icat.24486736e.2023.21.3.1860
“Once the refined search string was applied, primary studies were found and, in order to maximize the completeness of the search, the references of these studies were reviewed with the aim of identifying other relevant studies (a search technique known as "snowballing" (Wohlin, 2014).”
MethodInnovative Methods and Tools for Quality and Sustainability Management in Projects
Soares, T. D. O., & de Souza, J. P. E. (2023) · Springer eBooks · doi:10.1007/978-3-031-36121-0_80
An Anticipatory Algorithm for the Reviewer Assignment Issue using Machine learning Methods
Dubey, B. P. (2018) · Türk bilgisayar ve matematik eğitimi dergisi · doi:10.17762/turcomat.v9i2.13852
Digital Asset Production Using Lean Design Management: A Conceptual Framework
Karaz, M., & Teixeira, J. M. C. (2023) · Digital innovations in architecture, engineering and construction · doi:10.1007/978-3-031-30247-3_6
Human-centered algorithmic composition for well-being music
Junior, C. B. G., & Néris, V. P. D. A. (2023) · doi:10.5753/sbcas.2023.229409
Educational Data Comics: What can Comics do for Education in Visualization?Preprint
Boucher, M., Bach, B., Stoiber, C., et al. (2023) · doi:10.31219/osf.io/7fdb5
Klemmer, J. H., Gutfleisch, M., Stransky, C., Acar, Y., Sasse, M. A., & Fahl, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.00744
Towards adaptative monitoring for self-adaptative systems
de Catalunya, U. P., & Zavala, E. (2019) · UPCommons institutional repository (Universitat Politècnica de Catalunya) · doi:10.3926/phd-zavala · Cited by 2
Nabi, F. (2023) · Research Square · doi:10.21203/rs.3.rs-3355942/v1
González, E. A. S. (2023) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/196689
On the Understanding of the Role of Continuous Experimentation in Technology-Based Startup
Bernardes, M., & Marczak, S. D. S. (2023) · Brazilian Symposium on Software Engineering · doi:10.1145/3613372.3613414 · Cited by 1
“This systematic review followed the Snowballing procedure defined by Wohlin [39] and found 33 papers published on the topic of CE in SE.”
MethodHow Did Covid-19 Impact Software Design Activities in Global Software EngineeringPreprint
Faisal, N. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4588266
Together with library to professional and scientific manuscript
Bogojević, T., & Kermeci, D. (2022) · Korak biblioteke casopis za kulturu i bibliotecko-informacionu delatnost · doi:10.5937/korbib2207120b
A Scoping Review of Melon Manis Terengganu Research Perspective in Malaysia
Jusoh, M. F., Bakar, T. H. S. T. A., Abdullah, F. A., Maidin, M. K. H., & Muttalib, M. F. A. (2022) · Journal Of Agrobiotechnology · doi:10.37231/jab.2022.13.2.294 · Cited by 1
“The references from all records were examined to find untraced articles through forward and backward snowball searching Wohlin (2014).”
MethodANÁLISE DO AGRONEGÓCIO SOB A PERSPECTIVA DA ECONOMIA CIRCULAR: UMA REVISÃO SISTEMÁTICA DA LITERATURA
Kodama, T. K., MIYATA, H. H., Campoli, J. S., et al. (2023) · Anais ... Encontro Nacional de Engenharia de Produção/Anais do Encontro Nacional de Engenharia de Produção · doi:10.14488/enegep2023_tn_st_407_2007_46714
Competencies for Quality Management: A Systematic Literature Review and Perspectives for the Future
Brito, A. H. D. M., & Berssaneti, F. T. (2023) · International Conference on Industrial Engineering and Operations Management (IEOM) · doi:10.46254/eu6.20230121
Mapeo de los factores humanos relacionados con la realidad virtual: VRUX
Escallada, O., Lasa, G., Ugarte, M., Lammi, M., & Mazmela, M. (2023) · Proceedings · doi:10.61547/3488
Cross-disciplinary Architecting of Systems of Systems - Bridging the Gap
Cadavid, H. (2023) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.822663259
Andersen, M. B. (2023) · VBN Forskningsportal (Aalborg Universitet) · doi:10.54337/aau561775934 · Cited by 1
Feng, Z., Kimura, K., Trinkenreich, B., Sarma, A., & Steinmacher, I. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4632896
The Use of Circular Economy Indicators to Improve Sustainability in the Recycling Aluminium Context
Gonçalves, P. V. D. S., BARBOSA, R. F. M., Raspini, V. A. P., Sehnem, S., & Campos, L. M. D. S. (2021) · International Conference on Industrial Engineering and Operations Management (IEOM) · doi:10.46254/sa02.20210506 · Cited by 1
“Thus, according to Wohlin (2014), the step called start set identifies papers that started the snowballing method.”
BackgroundMethodComputational algebra system aimed at visually impaired people
Campoverde, P. H. M. (2022) · doi:10.47749/t/unicamp.2022.1254043
Warman, D., Tien, D., & Kabir, M. A. (2023) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc58545.2023.10327991
“This literature review has utilised the snowballing approach to generate the highly thorough list of literature surrounding this topic [2].”
MethodHRM Practice and Various Performance in SMEs: Systematic Literature Review
Pratama, C. B. A., & Soetjipto, B. E. (2023) · Advances in economics, business and management research/Advances in Economics, Business and Management Research · doi:10.2991/978-94-6463-302-3_31
Do Developers Fix Continuous Integration Smells?
Yaşa, A., Ergül, E., Erdogmus, H., et al. (2023) · doi:10.1145/3617555.3617870
“Following the snowballing approach suggested by Wohlin [16], we initiated our literature review with three studies [5, 6, 19], which ultimately yielded 18 academic papers.”
MethodHurricane: a Dataflow-oriented Data Service for Smart Cities Applications
Banni, M., Falci, M. L., Rosseti, I., & de Oliveira, D. (2023) · Journal of Information and Data Management · doi:10.5753/jidm.2023.3189
“In this article, we have chosen the snowballing strategy [Wohlin, 2014].”
MethodToward Leveraging Artificial Intelligence to Support the Identification of Accessibility Challenges
Aljedaani, W. (2023) · University of North Texas Digital Library (University of North Texas) · doi:10.12794/metadc2137559 · Cited by 2
An experiment on software test creation: can EvoSuite help test teaching?
Santos, L. M. L., Sousa, N., Gomes, L., Garcia, J., & Ferreira, F. (2023) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3629479.3629499
Privacy, security, and usability of mobile video conferencing applications: A comparative study
Samrgandi, N. H. (2023) · International Journal of ADVANCED AND APPLIED SCIENCES · doi:10.21833/ijaas.2023.11.014 · Cited by 1
Corrêa, C. M., & Silveira, M. S. (2023) · Simpósio Brasileiro de Fatores Humanos em Sistemas Computacionais (IHC) · doi:10.5753/ihc_estendido.2023.233072
Uma abordagem para apoiar a identificação de não executabilidade no teste estrutural de software
Neto, J. C. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2023.tde-19122023-151021
McKenzie, T., Hoermann, S., Lukosch, S. G., & Morales, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4669233 · Cited by 1
Sustainability of Systematic Literature Reviews
dos Santos, V. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2023.tde-20122023-111616
A Framework for Implementing National Respiratory Guidelines in Wales
Jefferies, R. (2023) · Cronfa (Swansea University) · doi:10.23889/suthesis.65359
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.