The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Active learning for screening prioritization in systematic reviews - A simulation study
Open accessAbstract
BackgroundConducting a systematic review requires great screening effort. Various tools have been proposed to speed up the process of screening thousands of titles and abstracts by engaging in active learning. In such tools, the reviewer interacts with machine learning software to identify relevant publications as early as possible. To gain a comprehensive understanding of active learning models for reducing workload in systematic reviews, the current study provides a methodical overview of such models. Active learning models were evaluated across four different classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two different feature extraction strategies (TF-IDF and doc2vec). Moreover, models were evaluated across six systematic review datasets from various research areas to assess generalizability of active learning models across different research contexts. MethodsPerformance of the models were assessed by conducting simulations on six systematic review datasets. We defined desirable model performance as maximizing recall while minimizing the number of publications needed to screen. Model performance was evaluated by recall curves, WSS@95, RRF@10, and ATD. ResultsWithin all datasets, the model performance exceeded screening at random order to a great degree.The models reduced the number of publications needed to screen by 91.7% to 63.9%. ConclusionsActive learning models for screening prioritization show great potential in reducing the workload in systematic reviews. Overall, the Naive Bayes + TF-IDF model performed the best.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These defaults were chosen on the basis of their consistently high performance in benchmark experiments across several datasets 31 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [31]
In the citing paper
“Their findings suggested that the NB + TF-IDF learning algorithm showed the highest estimated time savings and work savings oversampling, followed by the LR + TF-IDF, SVM + TF-IDF, and RF + TF-IDF learning algorithms.”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Ferdinands et al. (2020)
In the citing paper
“Additionally, the current literature lacks comprehensive comparative studies on the performance of learning algorithms when applied to real-world data (Burgard & Bittermann, 2023), and the impact of data characteristics on model performance is poorly understood (Ferdinands et al., 2020).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Ferdinands et al., 2020
In the citing paper
“The authors conducted a study that evaluated the performance of learning algorithms using four classifiers (LR, SVM, NB, and RF) and two feature extraction techniques (TF-IDF and D2V) on a set of six labeled datasets from systematic reviews using the software ASReview.”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Ferdinands et al. (2020)
In the citing paper
“However, this study had a limited sample size and did not explore the correlation between model performance and data characteristics, such as the proportion of relevant publications.”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Ferdinands et al. (2020)
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Distributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Bergstra, J., Yamins, D. L. K., & Cox, D. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 1,852
The Optimality of Naive Bayes.
Zhang, H. (2004) · Cited by 1,550
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 1×
“From the field of computer science, the Software dataset contains publications from a review on fault prediction in software engineering [51].”
Cited by
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Self-citationArtificial intelligence in systematic reviews: promising when appropriately used
van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072254 · Cited by 256
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Utrecht University Repository (Utrecht University) · Cited by 106
Self-citationScreening articles for systematic reviews with ChatGPT
Syriani, E., David, I., & Kumar, G. (2024) · Journal of Computer Languages · doi:10.1016/j.cola.2024.101287 · Cited by 82
Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09862-5 · Cited by 75
“An attempt to 19 Page 6 of 30 address the previous research gaps was presented by Ferdinands et al. (2020).”
Metadata
- Authors
- Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R.
- ORCID
- Publisher
- Center for Open Science
- Published
- 2020-09-16
- OpenAlex
- W3139098936
- Cited by
- 51
- References
- 50
- Citations by year
- 2026: 4 · 2025: 12 · 2024: 16 · 2023: 13 · 2022: 2 · 2021: 3 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- RePEc: Research Papers in Economics (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- active learning (machine learning), systematic review screening, machine learning classification, title and abstract screening, naive Bayes, TF-IDF, Doc2Vec, simulation study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Active learning for screening prioritization in systematic reviews - A simulation study. https://app.cochat.ai/papers/10.31219/osf.io/w6qbg
- The paper
- Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020). Active learning for screening prioritization in systematic reviews - A simulation study. https://doi.org/10.31219/osf.io/w6qbg
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.