Active learning for screening prioritization in systematic reviews - A simulation study
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
BackgroundConducting a systematic review requires great screening effort. Various tools have been proposed to speed up the process of screening thousands of titles and abstracts by engaging in active learning. In such tools, the reviewer interacts with machine learning software to identify relevant publications as early as possible. To gain a comprehensive understanding of active learning models for reducing workload in systematic reviews, the current study provides a methodical overview of such models. Active learning models were evaluated across four different classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two different feature extraction strategies (TF-IDF and doc2vec). Moreover, models were evaluated across six systematic review datasets from various research areas to assess generalizability of active learning models across different research contexts. MethodsPerformance of the models were assessed by conducting simulations on six systematic review datasets. We defined desirable model performance as maximizing recall while minimizing the number of publications needed to screen. Model performance was evaluated by recall curves, WSS@95, RRF@10, and ATD. ResultsWithin all datasets, the model performance exceeded screening at random order to a great degree.The models reduced the number of publications needed to screen by 91.7% to 63.9%. ConclusionsActive learning models for screening prioritization show great potential in reducing the workload in systematic reviews. Overall, the Naive Bayes + TF-IDF model performed the best.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These defaults were chosen on the basis of their consistently high performance in benchmark experiments across several datasets 31 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [31]
In the citing paper
“Their findings suggested that the NB + TF-IDF learning algorithm showed the highest estimated time savings and work savings oversampling, followed by the LR + TF-IDF, SVM + TF-IDF, and RF + TF-IDF learning algorithms.”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Ferdinands et al. (2020)
In the citing paper
“Additionally, the current literature lacks comprehensive comparative studies on the performance of learning algorithms when applied to real-world data (Burgard & Bittermann, 2023), and the impact of data characteristics on model performance is poorly understood (Ferdinands et al., 2020).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Ferdinands et al., 2020
In the citing paper
“The authors conducted a study that evaluated the performance of learning algorithms using four classifiers (LR, SVM, NB, and RF) and two feature extraction techniques (TF-IDF and D2V) on a set of six labeled datasets from systematic reviews using the software ASReview.”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Ferdinands et al. (2020)
In the citing paper
“However, this study had a limited sample size and did not explore the correlation between model performance and data characteristics, such as the proportion of relevant publications.”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Ferdinands et al. (2020)
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Distributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Bergstra, J., Yamins, D. L. K., & Cox, D. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 3,163
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · cited 1×
“For an example see the study by Wynants et al. [38].”
The Optimality of Naive Bayes.
Zhang, H. (2004) · Cited by 1,550
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 1×
“From the field of computer science, the Software dataset contains publications from a review on fault prediction in software engineering [51].”
A Survey of Text Classification Algorithms
Aggarwal, C. C., & Zhai, C. (2012) · Mining Text Data · doi:10.1007/978-1-4614-3223-4_6 · Cited by 842
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 2×
“Typically, only a fraction of the publications belong to the relevant class (2.94%, [4]).”
InfluentialThe GRoLTS-Checklist: Guidelines for Reporting on Latent Trajectory Studies
van de Schoot, R., Sijbrandij, M., Winter, S. D., Depaoli, S., & Vermunt, J. K. (2017) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2016.1247646 · Cited by 721 · cited 2×
“From the same research area, the PTSD dataset contains publications on studies applying latent trajectory analyses on posttraumatic stress after exposure to traumatic events [36].”
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 5×
“To draw conclusions about the general effectiveness of active learning models, it is essential to evaluate models on reviews from varying research contexts [7, 31].”
InfluentialA comparative study of TF*IDF, LSI and multi-words for text classification
Zhang, W., Yoshida, T., & Tang, X. (2011) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.08.066 · Cited by 676
Settles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.2200/s00429ed1v01y201207aim018 · Cited by 605
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532
Harrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 5×
“WSS indicates the reduction in publications needed to be screened, at a given level of recall [32].”
InfluentialSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 5×
“Recent studies have demonstrated the effectiveness of screening prioritization by means of active learning models [10, 11, 12, 13, 14, 15, 16].”
InfluentialActive learning with support vector machines
Kremer, J., Pedersen, K. S., & Igel, C. (2014) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1132 · Cited by 201
Reducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190 · cited 6×
“The suggestion that active learning is more difficult for datasets from the social sciences compared to data from the medical sciences [12] does not seem to be the case.”
InfluentialRathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184 · cited 1×
“A possible explanation for this is that this difficulty has to be attributed to factors more directly related to the systematic review at hand, such as the proportion of relevant publications or the complexity of inclusion criteria used to identify relevant publications [16, 54].”
Text Categorization Models for High-Quality Article Retrieval in Internal Medicine
Aphinyanaphongs, Y. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1641 · Cited by 177 · cited 1×
“It was found [26, 27] that different classification techniques can serve different needs in the retrieval of relevant publications, for example the desired balance between recall and precision.”
Shemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Using machine learning to advance synthesis and use of conservation and environmental evidence
Cheng, S. H., Augustin, C., Bethel, A., Gill, D., Anzaroot, S., Brun, J., DeWilde, B., Minnich, R. C., Garside, R., Masuda, Y. J., et al. (2018) · Conservation Biology · doi:10.1111/cobi.13117 · Cited by 155
Evaluation of machine-learning protocols for technology-assisted review in electronic discovery
Cormack, G. V., & Grossman, M. R. (2014) · Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval · doi:10.1145/2600428.2609601 · Cited by 149 · cited 3×
“Plotting recall as a function of the proportion of screened publications offers insight in model performance throughout the entire screening process [11, 13].”
Gates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148 · cited 4×
“Lastly, previous studies have mainly focussed on reviews from a single scientific field, like medicine [15, 16] or computer science [10, 11].”
InfluentialPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148 · cited 1×
“…[10, 11, 12, 13, 14, 15, 19, 20, 21], ready-to-use software tools implementing such models (Abstrackr [22], Colandr [23], FASTREAD [11], Rayyan [24], and RobotAnalyst [25]) currently use the same classification technique to predict relevance of publications, namely support vector machines (SVM).”
A benchmark and comparison of active learning for logistic regression
Yang, Y., & Loog, M. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.06.004 · Cited by 137
Text Similarity in Vector Space Models: A Comparative Study
Shahmirzadi, O., Lugowski, A., & Younge, K. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00120 · Cited by 126
Towards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence
Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L., & Haynes, R. B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2996 · Cited by 101
Virus Metagenomics in Farm Animals: A Systematic Review
Kwok, K. T. T., Nieuwenhuijse, D. F., Phan, M. V. T., & Koopmans, M. P. G. (2020) · Viruses · doi:10.3390/v12010107 · Cited by 97
Cross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88 · cited 2×
“A well-established approach to increase the efficiency of title and abstract screening is screening prioritization [8, 9].”
Finding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83 · cited 9×
“…forms and shapes [10, 11, 12, 13, 14, 15, 19, 20, 21], ready-to-use software tools implementing such models (Abstrackr [22], Colandr [23], FASTREAD [11], Rayyan [24], and RobotAnalyst [25]) currently use the same classification technique to predict relevance of publications, namely support vector…”
InfluentialFAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80 · cited 6×
“Recent studies have demonstrated the effectiveness of screening prioritization by means of active learning models [10, 11, 12, 13, 14, 15, 16].”
InfluentialAutonomy and Reliability of Continuous Active Learning for Technology-Assisted Review
Cormack, G. V., & Grossman, M. R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06868 · Cited by 62
ASReview LAB - A tool for AI-assisted systematic reviews
developers, A. L. (2026) · Zenodo · doi:10.5281/zenodo.3345592 · Cited by 59
Self-citationAppenzeller‐Herzog, C., Mathes, T., Heeres, M. L. S., Weiss, K. H., Houwen, R. H. J., & Ewald, H. (2019) · Liver International · doi:10.1111/liv.14179 · Cited by 51
Nudging healthcare professionals towards evidence-based medicine: A systematic scoping review
Nagtegaal, R., Tummers, L., Noordegraaf, M., & Bekkers, V. (2019) · Journal of Behavioral Public Administration · doi:10.30636/jbpa.22.71 · Cited by 45
Self-citationSemi-Automated evidence synthesis in health psychology: current methods and future prospects
Marshall, I. J., Johnson, B. T., Wang, Z., Rajasekaran, S., & Wallace, B. (2020) · Health Psychology Review · doi:10.1080/17437199.2020.1716198 · Cited by 34
Editorial: Systematic review automation thematic series
Lau, J. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0974-z · Cited by 27 · cited 1×
“Currently, the demand for systematic reviews exceeds the available time and resources by far [5].”
ASReview: Open Source Software for Efficient and Transparent Active Learning for Systematic Reviews.Preprint
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2020) · arXiv (Cornell University) · Cited by 21
Self-citationCertainty-Enhanced Active Learning for Improving Imbalanced Data Classification
Fu, J. H., & Lee, S. L. (2011) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2011.43 · Cited by 18
ASReview: Active learning for systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4647608 · Cited by 12
Self-citationImproving Active Learning in Systematic Reviews
Singh, G., Thomas, J., & Shawe-Taylor, J. (2018) · arXiv · doi:10.48550/arxiv.1801.09496 · Cited by 11
Text classification on imbalanced data: Application to systematic reviews automation
Ma, Y. (2007) · uO Research (University of Ottawa) · doi:10.20381/ruor-12126 · Cited by 10
Appenzeller‐Herzog, C. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3625931 · Cited by 8
Comparing Word Embeddings for Document Screening based on Active Learning.
Carvallo, A., & Parra, D. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · Cited by 6
Appenzeller‐Herzog, C. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3625930 · Cited by 5
Nudging healthcare professionals towards evidence-based medicine: A systematic scoping review
Nagtegaal, R., Tummers, L. G., Noordegraaf, M., & Bekkers, V. (2019) · EUR Research Repository (Erasmus University Rotterdam) · doi:10.7910/dvn/wmgpgz · Cited by 3
Self-citation
Cited by
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Self-citationArtificial intelligence in systematic reviews: promising when appropriately used
van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072254 · Cited by 256
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Utrecht University Repository (Utrecht University) · Cited by 106
Self-citationScreening articles for systematic reviews with ChatGPT
Syriani, E., David, I., & Kumar, G. (2024) · Journal of Computer Languages · doi:10.1016/j.cola.2024.101287 · Cited by 82
Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09862-5 · Cited by 75
“An attempt to 19 Page 6 of 30 address the previous research gaps was presented by Ferdinands et al. (2020).”
Metadata
- Authors
- Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R.
- ORCID
- Publisher
- Center for Open Science
- Published
- 2020-09-16
- OpenAlex
- W3139098936
- Cited by
- 51
- References
- 50
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 16 · 2023: 13 · 2022: 2 · 2021: 3 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- RePEc: Research Papers in Economics (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- active learning (machine learning), systematic review screening, machine learning classification, title and abstract screening, naive Bayes, TF-IDF, Doc2Vec, simulation study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.