Guidelines for snowballing in systematic literature studies and a replication in software engineering
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Abstract
Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.
Overview
CoChat Summary
TL;DR
The paper proposes a detailed snowballing procedure for systematic literature studies in software engineering and evaluates it by replicating a published systematic literature review. It concludes that snowballing can be a viable first search strategy and, in this case, found the same set of studies as the prior reviews.
Key findings
- The replication began with a tentative start set of 13 candidates; 3 were included in the start set.
- Across the study, 38 papers were evaluated and 11 papers were included in the final set.
- Efficiency varied sharply by step: start set 3/13 = 23%, iteration 1 7/25 = 28%, iteration 2 1/223 = 0.4%, and iteration 3 0%; overall efficiency was 3.7%.
- The paper states that the replication produced the same papers as the prior systematic literature reviews, and that 9 studies were in common with both reviews in [11].
Methods
- The paper develops a snowballing procedure based on experience from several systematic literature reviews and experimenting with different approaches; it specifies backward and forward snowballing, inclusion/exclusion handling, authors, and data extraction.
- For the replication, the author searched Google Scholar using the string “cross-company within-company software effort estimation” with time frame 1995–2005.
- The replication examined candidate papers through reference lists and citations, using title/publication-year screening, then abstracts, then full texts when needed.
Limitations
- The paper states a main validity threat: the researcher had read the original study 6–12 months earlier, which may have affected inclusion/exclusion decisions.
- The author notes that the identified start set was far from perfect, because the three start-set papers had one author in common, which could miss papers not linked to those papers.
- Evidence of limitations evident in the text: the efficiency results are based on a single replication case in one topic area, so generalizability beyond this case is not established (evident from the study design).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 3 Not valid · 11 Not checked · 131 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The same papers are identified.”
What the sources say
- Disputedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - sourceExperimental — accuracy not yet measured
The Q1.6 ‘Papers included’ section and the text following Table 1 say the two reviews selected ten primary studies each, but only nine were common to both. The paper describes citation analysis within those reviews, not a snowballing replication that identified exactly the same papers. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“MacDonell et al. evaluated the reliability of systematic reviews through comparing the results of two studies with a common research question performed by two independent groups of researchers.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Introduction says the paper assesses systematic-review reliability by comparing two independent reviews addressing the same research question. Section 3.1 specifies that two teams undertook those reviews independently, and the Conclusions report that the teams' outcomes were compared. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Nine studies are in common with both previous systematic literature reviews presented in [11].”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The passage following Table 1 states that both systematic literature reviews selected 10 primary studies, with nine in common. This directly supports the citing sentence. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If using the papers studied in detail as a efficiency measure, then the different approaches requires about the same effort.”
What the sources say
- Supportedcites [11] [11] MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676-687. - source
The Table 1 discussion says both reviews selected 10 primary studies, and §4.2 describes their data-extraction work on those studies. Thus, if the number of papers studied in detail is used as the efficiency measure, the two approaches are comparable on that measure; this does not establish equal time spent. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 155 claims and every source verdict.
- Probably not a substantial claim. 111
- It may state two separate claims as one. 60
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“we applied forward and backward snowballing based on the guidelines by Wohlin et al. [26]”
From A Multivocal Review of MLOps Practices, Challenges and Open Issues (2026) · cites this paper as Wohlin et al. [26]
The cited source says what the claim attributes to it.
In the citing paper
“As a guideline, this study was performed following well-known principles of systematic studies: References [193, 196, 207, 220].”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The cited source says what the claim attributes to it.
In the citing paper
“To minimize the variables that could compromise our study, we (i) conducted our study following accepted guidelines of systematic studies [181, 193, 207, 220], (ii) carefully designed our study in advance, and (iii) assessed, validated, and discussed the potential threats in each phase of the study.”
From Quantum Combinatorial Optimization in the NISQ Era: A Systematic Mapping Study (2024) · cites this paper as [220]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“HDFS logs describe operations on the storage block pool [22].”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“only 4 papers [19], [20], [22], [36] evaluate their techniques across different domains, training on a source domain and testing on a different target domain.”
From Practitioners’ expectations on log anomaly detection (2025) · cites this paper as [22]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449 · cited 7×
“Similar ideas have been brought into information systems research, e.g. by Webster and Watson [2].”
BackgroundMethodInfluentialSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 1×
“Systematic mapping studies have been highlighted as a complement to systematic literature reviews [8].”
BackgroundGreenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 4×
“Greenhalgh and Peacock [15] applied three different search methods in their research, and concluded that protocol-driven search approaches by themselves are not necessarily the most efficient method regardless of the number of databases used, since some sources may be found through personal knowledge/contacts (e.”
BackgroundResultInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 2×
“[13, 14] gives the impression that automation is better than manual search.”
BackgroundMethodGLOBAL ECONOMIC PROSPECTS: THE RECESSION MAY BE OVER BUT WHERE NEXT?
(2013) · The Financial Crisis and the Global South · doi:10.2307/j.ctt183pb3w.5 · Cited by 908
Building knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
Rodrik, D., & Velasco, A. (1999) · National Bureau of Economic Research · doi:10.3386/w7364 · Cited by 711
Leading issues in economic development
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2269 · Cited by 710
Systematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 11×
“Other complementary searches were not done either since the objective is to compare snowballing with a database-firstapproach in a similar way as in [19], although with a better specified procedure for using snowballing.”
BackgroundResultInfluentialSelf-citation
Cited by
Items 3,401–3,500 of 3,934
Text Mining Business Policy Documents: Applied Data Science in Finance
Spruit, M. R., & Ferati, D. (2023) · IGI Global eBooks · doi:10.4018/979-8-3693-2045-7.ch077
Disclosure of Environmental Compliance Management on Corporate Websites
Thimm, H. H., & Rasmussen, K. B. (2023) · IGI Global eBooks · doi:10.4018/979-8-3693-2045-7.ch025
Lemyre, F. C., Lévesque, S., Domingue, M., Herrmann, K., & Éthier, J. (2023) · medRxiv · doi:10.1101/2023.12.21.23300389
“Snowballing is generally used as a literature search method aiming at identifying papers belonging to a given field [54].”
MethodMobile Health from Developers’ Perspective
Oliveira, P., Costa, E., Andrade, R. M. C., Santos, I. S., & Neto, P. A. S. (2023) · SN Computer Science · doi:10.1007/s42979-023-02455-z
“Thus, after this empirical analysis, we decided to use the term “health” as the starting point to execute a snowballing [62] search for tags.”
[RETRACTED] Current State of Scaffolding Unsafe Factor: A Thematic Review
Mohammad, M. Z., & Jaafar, M. S. (2023) · International Journal of Sustainable Construction Engineering Technology · doi:10.30880/ijscet.2023.14.04.027 · Cited by 1
Spennemann, D. H. R. (2024) · International Journal of Building Pathology and Adaptation · doi:10.1108/ijbpa-04-2023-0039 · Cited by 2
A Systematic Mapping of Affective Intelligent Tutor Systems in Engineering Education
Figueira, T., Vieira, A. V. M., & Netto, J. F. D. M. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342953
Continual Object Detection with Deep Neural Networks
Menezes, A. G. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2023.tde-08012024-161234
A scoping review of auto-generating transformation between software development artifacts
Siahaan, D., Fauzan, R., Widyadhana, A., et al. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1306064
Universiti Teknologi, M. A. R. A., Razak, N. A. A., Ismail, Z., Universiti Teknologi, M. A. R. A., Brahim, J., & Universiti Teknologi, M. A. R. A. (2023) · Journal of Technology Management and Business · doi:10.30880/jtmb.2023.10.02.002
Uma Análise Abrangente de Soluções de Autenticação para Microsserviços: Uma Revisão Sistemática
Cargnelutti, R., Basso, F. P., & Bernardino, M. (2023) · Escola Regional de Engenharia de Software (ERES) · doi:10.5753/eres.2023.237314
de Souza, T. D. P. C., Lyra, K. T., Jaques, P. A., et al. (2023) · doi:10.5753/sbie.2023.234839
Albino, L. A. M., Rocha, S. O., Martines, G. B., Dell’Agli, B. A. V., & Caetano, L. M. (2024) · Contribuciones a las Ciencias Sociales · doi:10.55905/revconv.17n.1-393
RETRACTED: A Novel Software Product Manager’s Framework: A Systematic Literature Review
Parikh, N. A. (2024) · Open Journal of Business and Management · doi:10.4236/ojbm.2024.121036
RetractedA framework to support requirements validation in global software development
Minhas, N. M., Ahmad, B., & Nasir, N. (2023) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit60620.2023.00052
“To complement database searches, we adopted the snowball approach [34].”
MethodPenyebab dan Dampak Ekologis dari Susut Hasil Produksi Ikan di Indonesia
Budiman, I., Wisudyawati, D., & Azzahra, A. (2023) · Penerbit BRIN eBooks · doi:10.55981/brin.908.c755
Drolet, G., & Niquette, M. (2024) · Recherches féministes · doi:10.7202/1108765ar
Teixeira, R. (2024) · Revista Foco · doi:10.54751/revistafoco.v17n2-055
Nabi, F., Fatima, H., & Maple, C. R. (2024) · Research Square · doi:10.21203/rs.3.rs-3355942/v3
Knocking on knowledge’s door: mapping university to business marketing literature
Tusal, R. M. V., Álvarez, E. G., & Guzmán, J. S. (2024) · Journal of Marketing for HIGHER EDUCATION · doi:10.1080/08841241.2024.2323429
Nabi, F., & Hasan, R. (2024) · Research Square · doi:10.21203/rs.3.rs-3355942/v4
Evidence of validation in a strategy development model structure in public health schools
Reis, C. A. C. L., do Carmo, C. N., Filho, A. S. N., & de Souza, A. L. L. (2024) · F1000Research · doi:10.12688/f1000research.143843.1 · Cited by 1
Nabi, F., Hasan, R., & Maple, C. R. (2024) · Research Square · doi:10.21203/rs.3.rs-3355942/v5
Engagement Masyarakat dan Content Analysis terhadap Akun Instagram Lembaga Kementerian
Pangesti, D., & Hidayanto, A. N. (2024) · Syntax Literate Jurnal Ilmiah Indonesia · doi:10.36418/syntax-literate.v9i3.14876
“Pemilihan desain penelitian didasarkan pada pertanyaan penelitian atau permasalahan yang dibahas (Wohlin, 2014).”
BackgroundA Bibliometric Review of the Teaching Personal and Social Responsibility Model in Physical Education
Di, Y., Cui, L., Zhou, W., Wu, X., & Zhou, T. (2023) · International Conference on Information Science and Education (ICISE-IE) · doi:10.1109/icise-ie60962.2023.10456435
“It can help scholars to better find information on related topics and help scholars to find more related studies through literature snowballing [25].”
BackgroundRowell, D., McMillan, D., & Carroll, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4764238
Loufopoulos, G., Manaki, V., Tasoudis, P., et al. (2024) · medRxiv · doi:10.1101/2024.03.18.24304513
“We also searched the reference list of the included studies for potentially eligible studies using the snowball methodology 17 .”
MethodStakeholder Requirements for Big Data Projects: A Multi-Case Study ReviewPreprint
Regane, B., Power, N., Beecham, S., & Lemma, D. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4768913
Vitkos, E. N., Kounatidou, N. E., Konsolaki, E., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4763718
Fernandes, J. C., de Carvalho, L. S. G., de Oliveira, D. B. F., Oliveira, E., Pereira, F. D., & Lauschner, T. (2024) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2024.3587
Scripts of Whiteness in engineering: An analysis of the literature
Downey, R. G., Mejia, J. A., Chen, D. A., Hoople, G. D., & Lord, S. M. (2024) · Collaborative Network for Engineering and Computing Diversity Conference (CoNECD) · doi:10.18260/1-2--45475
Munisamy, M., Osman, S. Z. M., & Sanmugam, M. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-3124-8.ch005 · Cited by 2
Search Space Visualizations in the Context of Optimization Problems: A Literature Review
Silva, E. W. L., do Nascimento, H. A. D., Félix, J. P., et al. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-46549-9_6
Estimating Early Phase Agile Projects Using Functional Capability SizingPreprint
Rosa, W., & Jardine, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4805884
Auxiliando Professores de Engenharia de Requisitos a Identificar Jogos como Recursos Pedagógicos
Steglich, C., & Marczak, S. D. S. (2021) · Workshop em Engenharia de Requisitos (WER) · doi:10.29327/1298728.24-29
“O Estudo 2, já realizado, é uma revisão da literatura que teve como foco a identificação dos jogos que apoiam o ensino de requisitos, tendo a partir de um estudo inicial [20] a realização de um processo de forward snowballing [26], que consiste no seguinte processo: dado um conjunto inicial, observa-se quais estudos o…”
BackgroundA systematic literature review on the pharmacological applications of the carotenoid norbixin
Alves, A. L., Miranda, L. S., Alves, T. L., et al. (2024) · Peer Review · doi:10.53660/prw-2189-4028
New Algorithms and Benchmarks for Supporting Spectrum-Based Fault Localization
Vancsics, B. (2023) · SZTE Repository of Dissertations (University of Szeged) · doi:10.14232/phd.11603 · Cited by 1
Methods for Enhancing Software Fault Localization
Sarhan, Q. I. (2023) · SZTE Repository of Dissertations (University of Szeged) · doi:10.14232/phd.11772 · Cited by 1
Cybercrimes in the Darknet and Their Detections: A Comprehensive Analysis and Future Directions
Ali, H., Abdelrazek, M., Huda, S., Rashid, M., Rahman, M., & Ibrahim, A. S. (2024) · International journal of Web & Semantic Technology · doi:10.5121/ijwest.2024.15202
Charting the African Neuroethics Discourse: Ethical Issues of Neuroscience in AfricaPreprint
Oyeniji, O. A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4824538
Bukhsh, F. A., Govers, R. R., Bemthuis, R., et al. (2024) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197668122.013.20
Research Trends and Recommendations for Future Microservices Research
Stojanov, Ž., Hristoski, I. S., Stojanov, J., & Stojkov, A. (2024) · Proceedings of the Institute for System Programming of RAS · doi:10.15514/ispras-2024-36(1)-7 · Cited by 5
“• Ph#3: Using snowball search method [27] for finding additional studies and minimize the possibility of omitting relevant secondary studies.”
MethodZaguir, N. A. (2024) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.3.2024.tde-23052024-085756 · Cited by 2
Serverless Computing: A Systematic Literature Review of Challenges and OpportunitiesPreprint
Huang, C., Du, X., Huang, Y., Zhang, R., Yan, M., & Wu, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4845120
A Model for Dealing with Epistemic Uncertainties in Agile Software Project ManagementPreprint
Barbosa, J. F., de Farias, I. H., Marinho, M., & de Moura, H. P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4848407
Implementation Patterns for Zone Architectures in Enterprise-Grade Data Lakes
Giebler, C., Gröger, C., Hoos, E., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61057-8_16
From expert to flexpert. How workers meet new expertise needs throughout their careers
Frie, L. S. (2024) · Radboud University Press eBooks · doi:10.54195/9789493296305
Vitti, M., Sassanelli, C., & Facchini, F. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862262
Nabi, F., Hasan, R., & Maple, C. R. (2024) · Research Square · doi:10.21203/rs.3.rs-3355942/v6
User-Like Bots for Cognitive Automation: A SurveyPreprint
Hillmann, P., Gidey, H. K., Karcher, A., & Knoll, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862561
Experiments in thinking otherwise: Embracing tensions in designing sustainable futures
Neuhoff, R. (2024) · VBN Forskningsportal (Aalborg Universitet) · doi:10.54337/aau718335886 · Cited by 1
Wilson-Burke, E. (2024) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.26113699 · Cited by 2
Mathematical concepts in calculating Zakat Maal based on the Al-Qur'an perspective
Azizah, N., & Tohir, M. (2024) · Alifmatika Jurnal Pendidikan dan Pembelajaran Matematika · doi:10.35316/alifmatika.2024.v6i1.78-89
“A literature study guides studying a research problem (Kuhrmann et al., 2017; Okoli, 2015; Snyder, 2019; Torres-Carrión et al., 2018; Wohlin, 2014).”
BackgroundGupta, V. (2024) · Springer eBooks · doi:10.1007/978-3-031-58971-3_3
On Modularity of Neural Networks: Systematic Review and Open Challenges
Alho, R., Raatikainen, M., Myllyaho, L., & Nurminen, J. K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66459-5_2 · Cited by 2
Berrocoso, J. V. (2024) · RELATEC Revista Latinoamericana de Tecnología Educativa · doi:10.17398/1695-288x.23.2.1
Technologie als Wegbereiter: Schlüsselfaktoren der Implementierung von Hospital-at-Home
Maaser, Y., Salomo, S., & Wosch, S. (2024) · Springer eBooks · doi:10.1007/978-3-658-43860-9_20
Hunger, T., Arnold, M. G., Engesser, S., et al. (2024) · doi:10.31219/osf.io/qmg76
Managing everyday life: exploring the essential components of reablement and user experiences
Mouchaers, I. (2024) · Research Publications (Maastricht University) · doi:10.26481/dis.20240913im · Cited by 3
Doherty, G., Naylor, M. L., Draper, H., et al. (2024) · Preprints.org · doi:10.20944/preprints202408.1558.v1
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2024) · Springer eBooks · doi:10.1007/978-3-662-69306-3_2
Self-citationHe, X., Kozłowski, M., Ujang, N. B., & Ma, Y. (2024) · Open House International · doi:10.1108/ohi-03-2024-0107 · Cited by 3
Applications of Socially Aware Design in Education: a systematic mapping of the literature
Haddad, F. B. B., Corrêa, C. G., Filho, W. A. R., & Peres, L. M. (2024) · SBC Reviews on Computer Science · doi:10.5753/reviews.2024.3269
Getting into the game: gamifying software development with the GSA framework
Porto, D. D. P., Fabbri, S., & Ferrari, F. C. (2024) · Software Quality Journal · doi:10.1007/s11219-024-09694-0 · Cited by 1
“AsseenfromTable2,the stepsoftheWohlin(2014)’sprocessandthestagesoftheActionResearchprocesscanbe mappedontothestructureoftheGSAframework.”
InfluentialTrudel, P., Duval-Brassard, P., & Trottier, C. (2024) · Staps · doi:10.3917/sta.pr1.0102
Do quality and internationalization interrelate in higher education?
Opmane, K., Zeps, A., & Lapiņa, I. (2024) · Business and management · doi:10.3846/bm.2024.1313
Antonopoulos, I., Pechlivanidou, E., Olewnik, Ł. H., Zielińska, N., Chrysikos, D., Samolis, Α., Tsikouris, G., & Troupis, T. G. (2024) · Medicina · doi:10.3390/medicina60091504 · Cited by 1
“The references of each relevant article were further screened to find more eligible studies (the so-called “snowballing technique”) [35].”
BackgroundGroberio, J. G., Reginato, P. H., Streit, R. E., et al. (2024) · General Thoracic and Cardiovascular Surgery · doi:10.1007/s11748-024-02090-2
“Discrepancies were resolved in a panel discussion with the senior authors (W.F.G., A.Z.U., and F.R.F.) Additionally, we used backward snowballing (i.e., review of references) to identify relevant texts from articles identified in the original research [11].”
Embracing intersectional feminist thinking in design processes
Chopra, M., & Gaudio, C. D. (2024) · Journal of Design Business & Society · doi:10.1386/dbs_00061_1 · Cited by 2
Souza, C. D. S., Silveira, M. S., & Manssour, I. H. (2024) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia.2024.241929
Exploring Tools for Flaky Test Detection, Correction, and Mitigation: A Systematic Mapping Study
Martins, P. A. C., Alves, V. A., de Lima, I. K., Bezerra, C. I. M., & Machado, I. (2024) · doi:10.5753/sast.2024.3700
Mergers and acquisitions — The special case of German family firms: Research needs
Ulrich, P., & Stockert, F. (2024) · doi:10.22495/cgrapp27
“This is following the guidance set forth by Tranfield et al. (2003, p. 207) and Wohlin (2014, p. 3).”
MethodOn Complexity Classes of Envy-Free Pricing Problems: A Short Survey
Salvatierra, M. M., Salvatierra, M., Colonna, J. G., et al. (2024) · Operations Research Forum · doi:10.1007/s43069-024-00373-1
Ethical reasoning methods for ICT: What they are and when to use them
España, S., van der Maaten, C., Gulden, J., & Pastor, Ó. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102373
UMA REVISÃO SISTEMÁTICA SOBRE A AVALIAÇÃO DO LEAN HEALTHCARE
VASCONCELOS, R. A. X. D., Santos, L. C., Gohr, C. F., & Feit, G. (2024) · Anais ... Encontro Nacional de Engenharia de Produção/Anais do Encontro Nacional de Engenharia de Produção · doi:10.14488/enegep2024_tn_st_411_2014_47648
Schulte, R., & Kizilkan, K. (2024) · Zeitschrift für Klein- und Mittelunternehmen/Zeitschrift für KMU und Entrepreneurship · doi:10.3790/zfke.2024.1448702
Valenciano, I. C., Silva, J. V. D. A., de Rezende, R. L., et al. (2024) · doi:10.5753/eri-mt.2024.245827
“Este trabalho utilizou o método de mapeamento bibliográfico e a técnica snowballing proposta por Wohlin (2014), que se trata de um processo manual de coleta, seleção e análise de trabalhos científicos.”
Value Chain Analysis in the Film Industry: A Literature Review
Boediman, A. S., Pujawan, I. N., & Baihaqi, I. (2024) · KnE Social Sciences · doi:10.18502/kss.v9i32.17431 · Cited by 2
“Following the approach outlined by Wohlin [9], this study utilized snowballing to identify relevant research papers.”
Barba, M. C., & Sanz, R. S. S. (2024) · Revista de Investigación Científica de la UNF – Aypate · doi:10.57063/ricay.v2i2.47
Shuro, L., Lawrence, E. C., Knight, L., Schneider, H. M., & Tabana, H. (2024) · PLoS ONE · doi:10.1371/journal.pone.0311052 · Cited by 2
“In addition to participant selection, researchers will gather relevant documents from VillageReach, search databases and use a snowballing approach to identify articles and other relevant documents [44].”
The Value of Applying Big Data Analytics in Health Supply Chain Management
Nuaimi, D. A., & Awofeso, N. (2024) · F1000Research · doi:10.12688/f1000research.156525.2
Um Panorama das Tecnologias Inclusivas no Ensino de Programação para Alunos Surdos no Brasil
Soares, I. M. C., Prates, J. M., Murad, S. B., et al. (2024) · doi:10.5753/sbie.2024.242469
“A fim de incluir estudos adicionais relevantes a pesquisa, optou-se por adotar a estrat ´ egia de snowballing para frente ( forward ) [Wohlin 2014] na lista de refer ˆ encia dos artigos, refinando a busca e aumentando a cobertura da busca.”
Processos de Software para Jogos Digitais: uma Revisão Terciária da Literatura
Benigno, M. V. B., Teixeira, E. N., & Werner, C. M. L. (2024) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2024.242237 · Cited by 2
“O método forward snowballing (Wohlin, 2014) não foi utilizado por limitações temporais para realização da pesquisa.”
Evaluating low back pain through electrodermal activity: a systematic review protocol
Sebastiá-Amat, S., Tortosa‐Martínez, J., Sanchís-Soler, G., Hernández-Sánchez, S., & Cortell-Tormo, J. M. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-082814
Marchesani, F., Masciarelli, F., & Bikfalvi, A. (2024) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-75586-6_11
“Finally, to ensure that no articles were missed, we compute the cross-reference analysis also adopted in other SLR [32] by applying the backward and forward snowballing method to search into the deep of our sample.”
Peeters, R. (2024) · Research Publications (Maastricht University) · doi:10.26481/dis.20250125rp
Lota, L. N., Zaman, T. S., Azwad, M. M., Farah, L., Chowdhury, A., Anjum, Z., Islam, C., & Kamal, A. R. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5060080
Unraveling the Pain Points of Domain Modeling]{Unraveling the Pain Points of Domain ModelingPreprint
Sousa, I. V., Sales, T. P., Guerra, E. B., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5045287
Boediman, A. S., Pujawan, I. N., & Baihaqi, I. (2024) · doi:10.1109/isct62336.2024.10791120
Software Requirement Smells and Detection Techniques: A Systematic Literature Review
Alemneh, E., & Berhanu, F. (2024) · Cybernetics and Information Technologies · doi:10.2478/cait-2024-0037 · Cited by 2
Stettina, C. J., & Knoop, M. A. (2024) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice/itmc61926.2024.10794343
“In order to understand the current state of literature, we conducted a literature analysis following a snowball approach [13] using Google Scholar for academic sources, as well as a Google web search for grey literature using the initial search term “(agile OR scrum) outside (software OR IT)” .”
A systematic exploration of collaborative immersive systems for sense-making in STEM
Syiem, B. V., & Türkay, S. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2441963 · Cited by 3
“We used a combination of back-ward and forward citation snowballing (Wohlin 2014) and leveraged our contact with study authors of relevant papers (Moher et al. 2009).”
Teaching Literature Reviewing for Software Engineering Research
Baltes, S., & Ralph, P. (2024) · Springer eBooks · doi:10.1007/978-3-031-71769-7_19 · Cited by 2
Teaching Systematic Literature Reviews: Strategies and Best Practices
Genero, M., & Piattini, M. G. (2024) · Springer eBooks · doi:10.1007/978-3-031-71769-7_20
The Multidimensional Aspect of Technology Orders
Peixoto, R. T., de Leite, B. R. A., & de Melo, F. C. L. (2024) · Systems Research and Behavioral Science · doi:10.1002/sres.3126
Santos, S. H. N. (2024) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.55.2024.tde-06012025-164548
Gutiérrez, A. F. B. (2024) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/213434
Metadata
- Authors
- Wohlin, C.
- Affiliations
- Blekinge Institute of Technology (SE)
- Corresponding author
- Wohlin, Claes
- ORCID
- Wohlin, Claes
- Venue
- Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2014-05-13
- Pages
- 1-10
- OpenAlex
- W1975675278
- Cited by
- 4,234
- References
- 31
- Citations by year
- 2026: 433 · 2025: 552 · 2024: 651 · 2023: 594 · 2022: 566 · 2021: 477 · 2020: 356 · 2019: 250 · 2018: 175 · 2017: 98 · 2016: 60 · 2015: 17 · 2014: 4
- FWCI
- 50.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Funders
- Lunds Universitet
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- snowballing, systematic reviews, research guidelines, software engineering literature, replication study, literature search strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.