A Machine Learning Approach for Semi-Automated Search and Selection in Literature Studies
The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
Background. Search and selection of primary studies in Systematic Literature Reviews (SLR) is labour intensive, and hard to replicate and update. Aims. We explore a machine learning approach to support semi-automated search and selection in SLRs to address these weaknesses. Method. We 1) train a classifier on an initial set of papers, 2) extend this set of papers by automated search and snowballing, 3) have the researcher validate the top paper, selected by the classifier, and 4) update the set of papers and iterate the process until a stopping criterion is met. Results. We demonstrate with a proof-of-concept tool that the proposed automated search and selection approach generates valid search strings and that the performance for subsets of primary studies can reduce the manual work by half. Conclusions. The approach is promising and the demonstrated advantages include cost savings and replicability. The next steps include further tool development and evaluate the approach on a complete SLR.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Several statistical stopping metrics have been proposed in the literature [15, 21, 22, 35, 38, 49, 50, 53, 55]).”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [35]
In the citing paper
“Alternatively, heuristics have been proposed as a practical and effective way to define stopping rules (e.g., [7, 27, 35, 47]).”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [35]
In the citing paper
“These include computing an inflection point [16, 21, 35, 38, 49, 50], estimating recall for the sequential screening of a ranked list of references [22], or computing the lengths of consecutive spans of excluded documents that occur between each relevant document during screening [38, 53, 55].”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [35]
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
(2012) · Cited by 9,062
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,049
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Hierarchical Dirichlet Processes
Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000302 · Cited by 2,120
Systematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230 · cited 1×
“Systematic literature reviews (SLR) have become very popular in software engineering (SE) as a means to synthesize existing knowledge in a growing field of research. growing number of SLRs have even called for SLRs of SLRs [10, 15, 18], or tertiary studies.”
BackgroundA systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
Extended Boolean information retrieval
Salton, G., Fox, E. A., & Wu, H. (1983) · Communications of the ACM · doi:10.1145/182.358466 · Cited by 1,054
(2011) · Cited by 1,000
Evidence-Based Software Engineering and Systematic Reviews
Kitchenham, B. A., Budgen, D., & Brereton, P. (2015) · Chapman & Hall/CRC eBooks · doi:10.1201/b19467 · Cited by 759
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 1×
“Automated study selection by machine learning tools was also addressed in an SLR by medicine researchers O’Mara-Eves et al. [25]. find that tool support for study selection can be used for prioritising which papers to validate, and that automated study selection may be used with low loss of recall.…”
MethodSystematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 1×
“Compared to term-based search, there is some evidence that it performs similarly regarding recall, and some researchers even report improved precision [2, 14, 31].”
BackgroundIdentifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 557
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1×
“Medicine researchers Tsafnat et al. [33] performed a SLR on automation for SLRs, they describe in detail the potential for automation on the different parts of the SLR process.”
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Research synthesis in software engineering: A tertiary study
Cruzes, D. S., & Dybå, T. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.01.004 · Cited by 346
LIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
Experiences from using snowballing and database searches in systematic literature studies
Badampudi, D., Wohlin, C., & Petersen, K. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745818 · Cited by 221
On the reliability of mapping studies in software engineering
Wohlin, C., Runeson, P., da Mota Silveira Neto, P. A., Engström, E., do Carmo Machado, I., & de Almeida, E. S. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.04.076 · Cited by 206
Self-citationDeveloping search strategies for detecting relevant experiments
Dieste, O., Grimán, A., & Juristo, N. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9091-7 · Cited by 200 · cited 3×
“…not contain the full text of the papers, and is instead restricted to title, abstract, and keywords. is not necessarily a problem: Dieste et al. [11] showed for manual search strings, that using only title, abstract, and keywords gave better precision while sacrificing little recall compared to…”
BackgroundMethodReducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190 · cited 3×
“classified as positive by the classifier higher. weight is the ratio between negative and positive examples in the training set [21].”
BackgroundSix years of systematic literature reviews in software engineering: An updated tertiary study
da Silva, F. Q. B., Santos, A. L. M., Soares, S., França, A. C. C., Monteiro, C. V. F., & Maciel, F. F. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.04.004 · Cited by 166
Tools to Support Systematic Literature Reviews in Software Engineering: A Mapping Study
Marshall, C., & Brereton, P. (2013) · 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.32 · Cited by 87 · cited 1×
“In a mapping study and a feature analysis on SLR tools, Marshall et al. [19, 20] found that existing tools have poor support for automated selection.”
MethodAutomatic Evidence Retrieval for Systematic Reviews
Choong, M. K., Galgani, F., Dunn, A. G., & Tsafnat, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3369 · Cited by 84
Reference-based search strategies in systematic reviews
Skoglund, M., & Runeson, P. (2009) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2009.4 · Cited by 71
Self-citationToward modernizing the systematic review pipeline in genetics: efficient updating via data mining
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A. (2012) · Genetics in Medicine · doi:10.1038/gim.2012.7 · Cited by 62 · cited 2×
“As a consequence, updating SLR search may be done continuously, by rerunning the search and selection tool [34]. saved effort on search and selection may instead be spent on systematically synthesising the knowledge in the SLR; a research effort that is much needed as the amount of SE research…”
BackgroundMethodTools to support systematic reviews in software engineering: a feature analysis
Marshall, C., Brereton, P., & Kitchenham, B. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601270 · Cited by 56
Learning first order logic rules with a genetic algorithm
Augier, S., Venturing, G., & Kodratoff, Y. (1995) · Knowledge Discovery and Data Mining · Cited by 52
Olorisade, B. K., de Quincey, E., Brereton, P., & Andras, P. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2915982 · Cited by 42
Robinson, K. A., Dunn, A. G., Tsafnat, G., & Glasziou, P. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.11.015 · Cited by 41
Validating Search Processes in Systematic Literature Reviews
Kitchenham, B. A., Li, Z., & Burn, A. J. (2011) · International Workshop on Evidential Assessment of Software Technologies (EAST) · doi:10.5220/0003557000030009 · Cited by 31
A Federated Search Approach to Facilitate Systematic Literature Review in Software Engineering
Ghafari, M. (2012) · International Journal of Software Engineering & Applications · doi:10.5121/ijsea.2012.3202 · Cited by 27
jLDADMM: A Java package for the LDA and DMM topic modelsPreprint
Nguyen, D. Q. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.03835 · Cited by 15
Research works printed but not linked: 1
- S. Bird E. Klein and E. Loper. 2009. Natural Language Processing with Python -- Analyzing Text with the Natural Language Toolkit. O'Reilly Media Inc. S. Bird E. Klein and E. Loper. 2009. Natural Language Processing with Python -- Analyzing Text with the Natural Language Toolkit. O'Reilly Media Inc.
Cited by
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
Artificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170
“As suggested above, the selection of primary studies is strongly related with the quality of the search, so the first task could benefit from an automatic definition of search strings too [36].”
BackgroundInfluentialSuicide before and during the COVID-19 Pandemic: A Systematic Review with Meta-Analysis
Yan, Y., Hou, J., Li, Q., & Yu, N. X. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043346 · Cited by 185
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146
“Several statistical stopping metrics have been proposed in the literature [15, 21, 22, 35, 38, 49, 50, 53, 55]).”
BackgroundOfori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10844-w · Cited by 86
Metadata
- Authors
- Ros, R., Bjarnason, E., & Runeson, P.
- Affiliations
- Lund University (SE)
- Venue
- Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2017-06-15
- Pages
- 118-127
- OpenAlex
- W2621485502
- Cited by
- 80
- References
- 39
- Citations by year
- 2026: 12 · 2025: 12 · 2024: 16 · 2023: 14 · 2022: 4 · 2021: 7 · 2020: 2 · 2019: 6 · 2018: 7
- FWCI
- 10.89
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Lund University Publications (Lund University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- systematic reviews, automated search, text classification, snowballing, replicability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.