The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Successful combination of database search and snowballing for identification of primary studies in systematic literature studies
Open accessAbstract
A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which was later complemented with snowball searches. Our conjecture is that we can perform even better searches if combining these two search approaches, referred to as a hybrid search strategy. Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared four alternative hybrid search strategies to assess whether we could identify more cost-efficient ways of searching for relevant primary studies. To compare and evaluate the hybrid search strategy, we replicated the search procedure in a systematic literature review (SLR) on industry–academia collaboration in software engineering. The SLR used a more “traditional” approach to searching for relevant articles for an SLR, while our replication was executed using a hybrid search strategy. In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30% more primary studies and even more studies when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when performing systematic literature studies. The hybrid search strategy is a strong contender for being used when performing systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 12×
“Once a start set was identified, back-ward and forward snowballing were performed to determine further articles to include, using the guidelines in Wohlin (2014).”
InfluentialSelf-citationSystematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659
Self-citationIdentifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 557
A Model for Technology Transfer in Practice
Gorschek, T., Garre, P., Larsson, S., & Wohlin, C. (2006) · IEEE Software · doi:10.1109/ms.2006.147 · Cited by 350
Self-citationThe role of replications in Empirical Software Engineering
Shull, F. J., Carver, J. C., Vegas, S., & Juristo, N. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9060-1 · Cited by 315
Garousi, V., Petersen, K., & Ozkan, B. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.07.006 · Cited by 241
Experiences from using snowballing and database searches in systematic literature studies
Badampudi, D., Wohlin, C., & Petersen, K. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745818 · Cited by 221
Self-citationOn the reliability of mapping studies in software engineering
Wohlin, C., Runeson, P., da Mota Silveira Neto, P. A., Engström, E., do Carmo Machado, I., & de Almeida, E. S. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.04.076 · Cited by 206
Self-citationDeveloping search strategies for detecting relevant experiments
Dieste, O., Grimán, A., & Juristo, N. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9091-7 · Cited by 200
Mourão, E., Pimentel, J. F., Murta, L., Kalinowski, M., Mendes, E., & Wohlin, C. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106294 · Cited by 191
Self-citationThe Success Factors Powering Industry-Academia Collaboration
Wohlin, C., Aurum, A., Angelis, L., Phillips, L., Dittrich, Y., Gorschek, T., Grahn, H., Henningsson, K., Kagstrom, S., Low, G., et al. (2012) · IEEE Software · doi:10.1109/ms.2011.92 · Cited by 185
Self-citationGuidelines for the Search Strategy to Update Systematic Literature Reviews in Software Engineering
Wohlin, C., Mendes, E., Felizardo, K. R., & Kalinowski, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106366 · Cited by 147
Self-citationUsing Forward Snowballing to update Systematic Reviews in Software Engineering
Felizardo, K. R., Mendes, E., Kalinowski, M., de Souza, É. F., & Vijaykumar, N. L. (2016) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962630 · Cited by 123
Self-citationRefining the systematic literature review process—two participant-observer case studies
Kitchenham, B. A., Pearl Brereton, A., Turner, M., Niazi, M., Linkman, S., Pretorius, R., & Budgen, D. (2010) · Empirical Software Engineering · doi:10.1007/s10664-010-9134-8 · Cited by 120
How Reliable Are Systematic Reviews in Empirical Software Engineering?
MacDonell, S., Shepperd, M., Kitchenham, B., & Mendes, E. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.28 · Cited by 108
Self-citationAction research as a model for industry-academia collaboration in the software engineering context
Petersen, K., Gencel, Ç., Asghari, N., Baca, D., & Betz, S. (2014) · doi:10.1145/2647648.2647656 · Cited by 77
Investigating the Use of a Hybrid Search Strategy for Systematic Reviews
Mourao, E., Kalinowski, M., Murta, L., Mendes, E., & Wohlin, C. (2017) · 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1109/esem.2017.30 · Cited by 77
Self-citationSecond-generation systematic literature studies using snowballing
Wohlin, C. (2016) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2915970.2916006 · Cited by 76
Self-citationReference-based search strategies in systematic reviews
Skoglund, M., & Runeson, P. (2009) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2009.4 · Cited by 71
Porte, G. K. (2013) · CALICO Journal · doi:10.11139/cj.30.1.10-15 · Cited by 57
Achieving industrial relevance in software engineering education
Wohlin, C., & Regnell, B. (1999) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/csee.1999.755175 · Cited by 55
Self-citationDetermining the Impact of Software Engineering Research on Practice
(2008) · doi:10.1109/mc.2008.85 · Cited by 51
Empirical software engineering research with industry: Top 10 challenges
Wohlin, C. (2013) · Conducting Empirical Studies in Industry · doi:10.1109/cesi.2013.6618469 · Cited by 48
Self-citationConnor, A. M., Buchan, J., & Petrova, K. (2009) · International Conference on Information Technology: New Generations · doi:10.1109/itng.2009.134 · Cited by 38
The Science Game: An Introduction to Research in the Social Sciences
Kelly, J. M., Agnew, N. M., & Pike, S. W. (1987) · Teaching Sociology · doi:10.2307/1318041 · Cited by 36
Requirements engineering and industrial uptake
Morris, P. J., Masera, M., & andre, W. M. (1998) · Requirements Engineering · doi:10.1007/bf02919966 · Cited by 34
Empirical studies for innovation dissemination: ten years of experience
Baldassarre, M. T., Caivano, D., & Visaggio, G. (2013) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2460999.2461020 · Cited by 27
Search Strategy to Update Systematic Literature Reviews in Software Engineering
Mendes, E., Felizardo, K. R., Wohlin, C., & Kalinowski, M. (2019) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2019.00061 · Cited by 18
Self-citationEvaluating strategies for forward snowballing application to support secondary studies updates
Felizardo, K. R., Iwazaki, A. Y., de Souza, É. F., Vijaykumar, N. L., & Nakagawa, E. Y. (2018) · Brazilian Symposium on Software Engineering · doi:10.1145/3266237.3266240 · Cited by 12
Research works printed but not linked: 5
- Punter, 2010, Industrial impact and lessons learned, 279, 10.1007/978-90-481-9849-8_17 · doi:10.1007/978-90-481-9849-8_17
- Rombach, 2008, Impact of research on practice in the field of inspections, reviews and walkthroughs: Learning from successful industrial uses, SIGSOFT Softw. Eng. Notes, 33, 26, 10.1145/1449603.1449609 · doi:10.1145/1449603.1449609
- S.J. Lamprecht, G.-J. van Rooyen, Models for technology research collaboration between industry and academia in South Africa, in: Proceedings IEEE Software Engineering Colloquium, SE, 2012, pp. 11–17. · doi:10.1109/se.2012.6242350
- W. Raschke, M. Zilli, J. Loinig, R. Weiss, C. Steger, C. Kreiner, Embedding research in the industrial field: A case of a transition to a software product line, in: Proceedings International Workshop on Long–Term Industrial Collaboration on Software Engineering, 2014, pp. 3–8. · doi:10.1145/2647648.2647649
- P. Krishnan, K. Ross, P. Pari-Salas, Industry academia collaboration: an experience report at a small university, in: Proceedings Conference on Software Engineering Education and Training, 2009, pp. 117–121. · doi:10.1109/cseet.2009.22
Cited by
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“The hybrid approach is the combination of search strategy and snowballing [32, 33].”
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
Synergies and gaps between circularity assessment and Life Cycle Assessment (LCA)
Samani, P. (2023) · Science of The Total Environment · doi:10.1016/j.scitotenv.2023.166611 · Cited by 81
Sustainable software engineering: Reflections on advances in research and practice
Venters, C. C., Capilla, R., Nakagawa, E. Y., Betz, S., Penzenstadler, B., Crick, T., & Brooks, I. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107316 · Cited by 59
Metadata
- Authors
- Wohlin, C., Kalinowski, M., Romero Felizardo, K., & Mendes, E.
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2022-07
- Volume
- 147
- Pages
- 106908
- ISSN
- 0950-5849 (ISSN-L)
- OpenAlex
- W4220698983
- Cited by
- 432
- References
- 29
- FWCI
- 44.84
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Funders
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (312827/2020-2)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.