Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records
Open accessThe DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 28 of the 38 sources its claims cite were checked for support.
Abstract
BACKGROUND: Conducting a systematic review demands a significant amount of effort in screening titles and abstracts. To accelerate this process, various tools that utilize active learning have been proposed. These tools allow the reviewer to interact with machine learning software to identify relevant publications as early as possible. The goal of this study is to gain a comprehensive understanding of active learning models for reducing the workload in systematic reviews through a simulation study. METHODS: The simulation study mimics the process of a human reviewer screening records while interacting with an active learning model. Different active learning models were compared based on four classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two feature extraction strategies (TF-IDF and doc2vec). The performance of the models was compared for six systematic review datasets from different research areas. The evaluation of the models was based on the Work Saved over Sampling (WSS) and recall. Additionally, this study introduces two new statistics, Time to Discovery (TD) and Average Time to Discovery (ATD). RESULTS: The models reduce the number of publications needed to screen by 91.7 to 63.9% while still finding 95% of all relevant records (WSS@95). Recall of the models was defined as the proportion of relevant records found after screening 10% of of all records and ranges from 53.6 to 99.8%. The ATD values range from 1.4% till 11.7%, which indicate the average proportion of labeling decisions the researcher needs to make to detect a relevant record. The ATD values display a similar ranking across the simulations as the recall and WSS values. CONCLUSIONS: Active learning models for screening prioritization demonstrate significant potential for reducing the workload in systematic reviews. The Naive Bayes + TF-IDF model yielded the best results overall. The Average Time to Discovery (ATD) measures performance of active learning models throughout the entire screening process without the need for an arbitrary cut-off point. This makes the ATD a promising metric for comparing the performance of different models across different datasets.
Overview
CoChat Summary
TL;DR
This simulation study compared seven active-learning models across six systematic review datasets and introduced Average Time to Discovery (ATD) as a screening-prioritization metric. It found substantial workload reductions, with NB + TF-IDF generally performing best and ATD ranking models similarly to recall and WSS@95.
Key findings
- Across models and datasets, WSS@95 ranged from 63.9% to 91.7%, meaning models reduced the number of publications needed to screen while still finding 95% of relevant records.
- Recall after screening 10% of records (RRF@10) ranged from 53.6% to 99.8%; ATD values ranged from 1.4% to 11.7%, indicating the average proportion of labeling decisions needed to detect a relevant record.
- NB + TF-IDF yielded the best results overall: it ranked first in ACE, Virus, and Wilson, shared first in PTSD and Software, and second in Nudging. RF + TF-IDF never performed best on any measure across datasets.
- ATD values showed a similar ranking across simulations as recall and WSS values, and all models exceeded random screening.
Methods
- Simulation study mimicking human title/abstract screening with active learning; seven models combined four classifiers (naive Bayes, logistic regression, support vector machines, random forest) with two feature-extraction strategies (TF-IDF, doc2vec), excluding NB + D2V because multinomial NB requires positive feature values.
- Six labeled systematic review datasets from medicine, computer science, and social science were used, yielding 42 simulations. Each simulation started with one relevant and one irrelevant record, retrained after each label, ended after all records were labeled, and was repeated 15 times with random initial training sets held constant within dataset.
- Hyperparameters were optimized via Tree of Parzen Estimators; simulations used ASReview 0.9.3 and analyses used R 3.6.1. Performance metrics were WSS@95, RRF@10, Time to Discovery (TD), and Average Time to Discovery (ATD), with recall curves, means with standard deviations, medians with MAD, and 95% confidence intervals.
Limitations
- Stated: hyperparameters were optimized for each model-dataset combination, so results reflect maximum model performance and generalizability to datasets without optimized hyperparameters remains unknown; the sample size of six datasets does not allow investigating how data characteristics relate to model performance.
- Stated: runtime varied widely across models, with implications for practical use because an efficient model should keep up with reviewer decision speed; ATD is not set against a baseline, and more research is needed on its statistical bias, outliers, sparse data, sensitivity to dataset characteristics, and construct/face validity.
- Evident: the study is a simulation using existing labels rather than real reviewer interaction, and the six datasets are a convenience sample; D2V versus TF-IDF results were inconclusive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




![Algorithm 3 Pseudo code for balancing a labeled dataset by Dynamic Resampling. Hyperparameters are a, that governs the weight of the 1’s; α, that governs the scaling the weight of the 1’s, as a function of the ratio of ones to zeros; b, governs how strongly we want to sample depending on the total number of samples; β, governs the scaling of the weight of the zeros depending on the number of samples. Values for these hyperparameters can be found on the GitHub repository of this paper [37]](https://figures.cochat.ai/paper-figures/207d3b30de3f42c29820726df524edc5/performance-of-active-learning-models-for-screening-prioritization-in-systematic-figure-5.jpg)
Verification
Claims the paper makes
15 Valid · 2 Not valid · 3 Not checked · 70 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“ready-to-use software tools implementing such models (Abstrackr [22], Colandr [23], FASTREAD [11], Rayyan [24], and RobotAnalyst [25]) currently use the same classification technique to predict relevance of publications, namely support vector machines (SVM).”
What the sources say
- Not checked — no full text of the sourcecites [22] Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. In: Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. IHI ’12. Association for Computing Machinery; 2012. p. 819–824. https://doi.org/10.1145/2110363.2110464. source
- Disputedcites [23] Cheng SH, Augustin C, Bethel A, Gill D, Anzaroot S, Brun J, et al. Using machine learning to advance synthesis and use of conservation and environmental evidence. Conserv Biol. 2018;32(4):762–4. https://doi.org/10.1111/cobi.13117. sourceExperimental — accuracy not yet measured
The paper describes Colandr as a tool that learns from users’ relevance decisions, but identifies its relevance-ranking method as word2vec, not SVM (Abstract, page not specified in our copy). It does not establish what classification techniques the other named tools use. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
- Supportedcites [11] Yu Z, Kraft NA, Menzies T. Finding better active learners for faster literature reviews. Empir Softw Eng. 2018;23(6):3161–86. https://doi.org/10.1007/s10664-017-9587-0. source
The citation is attached to FASTREAD, not to the other tools in the list. Finding 2 (p. 14) identifies FASTREAD as the method developed in this paper, and §5.4 (p. 12) states that all tested treatments use a linear SVM classifier. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Screening prioritization via active learning allows for substantial time-saving as the reviewer can decide to stop screening after a sufficient number of relevant publications have been found [7, 10].”
What the sources say
- Partly supportedcites [7] Cohen AM, Ambert K, McDonagh M. Cross-topic learning for work prioritization in systematic review creation and update. J Am Med Inform Assoc. 2009;16(5):690–704. sourceExperimental — accuracy not yet measured
The Conclusions (page unavailable) support prioritizing likely includable articles to make better use of reviewers’ time. They propose screening the remaining citations later, not stopping once enough relevant publications have been found. The Results distinguish this approach from active learning and describe active learning as less attractive for systematic-review work prioritization because the system, rather than reviewers, chooses which articles are read first. (LLM, conf=0.86)
- Supportedcites [10] Yu Z, Menzies T. FAST2: an intelligent assistant for finding relevant papers. Expert Syst Appl. 2019;120:57–71. https://doi.org/10.1016/j.eswa.2018.11.021. source
The Abstract (p. 1) says FAST2 reduces the effort needed to find most relevant papers and estimates how many remain, helping the reviewer decide when to stop. Section 4.2 (p. 12) describes a stopping target of 95% recall, consistent with stopping after a sufficient number—not necessarily all—relevant papers have been found. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In a previous study, the ACE dataset was used to simulate a model that did not use active learning, finding a WSS@95 value of 56.61% [31]”
What the sources say
- Supportedcites [31] Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc. 2006;13(2):206–19. https://doi.org/10.1197/jamia.M1929. source
Table 6 (Results; page not provided in our copy) reports WSS@95% of 56.61% for the ACEInhibitors review dataset. The Evaluating the Classifier section describes cross-validation of a classifier, consistent with the cited non-active-learning comparison. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Typical for systematic reviewing is that only a fraction of the publications belong to the relevant class [4].”
What the sources say
- Supportedcites [4] Borah R, Brown AW, Capers PL, Kaiser KA. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open. 2017;7(2):e012545. source
The Results report that, in the systematic reviews analysed, a mean of 2.94% of publications found in the initial search were ultimately included. This supports the claim that only a fraction of retrieved publications belong to the relevant set, though the measured reviews concerned medical interventions. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Another advantage of this technique is its short computation time, see [51] for a detailed comparison of computation times.”
What the sources say
- Supportedcites [51] Teijema J, Hofstee L, Brouwer M, de Bruin J, Ferdinands G, de Boer J, et al. Active learning-based Systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders. PsyArXiv; 2022. https://doi.org/10.31234/osf.io/t7bpd. source
The Discussion states that TF-IDF has significantly lower computational time than SBERT. Table 1 provides the detailed comparison: TF-IDF feature extraction took seconds in the listed combinations, versus more than six hours for SBERT. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
50 more claims · 1 with their cited sources checked
Create a free account to see all 90 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 16
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“For last-to-find records [7,19,22], advanced models significantly outperform simpler models due to their ability to detect contextual and semantic relationships.”
From Large-scale simulation study of active learning models for systematic reviews (2025) · cites this paper as [19]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes ( > 2000 publications) where approximately less than 5% of publications are relevant 13,20 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [20]
Experimental — accuracy not yet measuredIn the citing paper
“While useful for evaluating performance at a fixed recall level, WSS@95% provides no insight into how a model performs in scenarios such as quickly retrieving relevant records after a cold start or identifying the last-to-find records [7,19,22].”
From Large-scale simulation study of active learning models for systematic reviews (2025) · cites this paper as [19]
In the citing paper
“So, ASReview has demonstrated its ability to speed up systematic reviews.26,27,28,29,30,31,32,33,34,35,36”
From ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts (2025) · cites this paper as [27]
In the citing paper
“At that point, metrics such as recall, loss, time to discovery,27 or work saved over sampling (WSS@95)53 can be calculated.”
From ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts (2025) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Datasets
“For the remaining six datasets, the proportions of relevant publications were centered around 1-2% (Table 1).”
§ Datasets
“The screening process was simulated on a convenience sample of six labeled datasets.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Abstract
“The models reduce the number of publications needed to screen by 91.7 to 63.9% while still finding 95% of all relevant records (WSS@95).”
§ Results
“| NB + TF-IDF | 71.7 (1.37) | 91.7 (0.27) | 92.3 (0.08) | 82.9 (0.99) | 71.2 (0.62) | 83.4 (0.89) |”
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · cited 1×
“While previous studies have evaluated active learning models in many forms and shapes [10–15, 17–19], ready-to-use software tools implementing such models (Abstrackr [22], Colandr [23], FASTREAD [11], Rayyan [24], and RobotAnalyst [25]) currently use the same classification technique to predict relevance of publicatio…”
MethodAn open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Self-citationA Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
A Survey of Text Classification Algorithms
Aggarwal, C. C., & Zhai, C. (2012) · Mining Text Data · doi:10.1007/978-1-4614-3223-4_6 · Cited by 842 · cited 1×
“These are widely adopted techniques in text classification [28] and are fit for software tools to be used in scientific practice due to their relatively short computation time.”
MethodBorah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 2×
“However, conducting a systematic review is a costly endeavor that often requires over a year of work by a team of researchers [4], and includes the manual screening of thousands of titles and abstracts.”
BackgroundThe GRoLTS-Checklist: Guidelines for Reporting on Latent Trajectory Studies
van de Schoot, R., Sijbrandij, M., Winter, S. D., Depaoli, S., & Vermunt, J. K. (2017) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2016.1247646 · Cited by 721
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 3×
“Moreover, the WSS makes assumptions about the acceptable recall level whereas this level might depend on the research question at hand [32].”
BackgroundInfluential
Research works printed but not linked: 5
- Carvallo A, Parra D. Comparing word embeddings for document screening based on active learning. In: BIRNDL@ SIGIR. 2019. p. 100–107.
- Le Q, Mikolov T. Distributed representations of sentences and documents. In: International conference on machine learning. Beijing: PMLR; 2014. p. 1188–1196.
- Bergstra J, Yamins D, Cox D. Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: International Conference on Machine Learning. 2013. p. 115–123. http://proceedings.mlr.press/v28/bergstra13.html. Accessed Mar 2020.
- Mitchell TM. Does Machine Learning Really Work? AI Mag. 1997;18(3):11.
- Kanoulas E, Li D, Azzopardi L, Spijker R. CLEF 2019 technology assisted reviews in empirical medicine overview. In: CEUR workshop proceedings, vol. 2380. Lugano: CEU; 2019.
Cited by
Artificial intelligence in systematic reviews: promising when appropriately used
van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072254 · Cited by 256
Evaluating the effectiveness of large language models in abstract screening: a comparative analysis
Li, M., Sun, J., & Tan, X. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02609-x · Cited by 125
Utilizing ASReview in screening primary studies for meta-research in SLA: A step-by-step tutorial
Quan, Y., Tytko, T., & Hui, B. (2024) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2024.100101 · Cited by 64
Bente, B. E., van Dongen, A., Verdaasdonk, R. M., & van Gemert-Pijnen, L. (2024) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1332707 · Cited by 79
Oude Wolcherink, M. J., Pouwels, X. G. L. V., van Dijk, S. H. B., Doggen, C. J. M., & Koffijberg, H. (2023) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2023.2234639 · Cited by 50
Metadata
- Authors
- Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R.
- Affiliations
- Utrecht University (NL)
- Corresponding author
- Ferdinands, Gerbrich
- ORCID
- Venue
- Systematic Reviews
- Published
- 2023-06-20
- Volume
- 12
- Issue
- 1
- Pages
- 100-100
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC10280866
- OpenAlex
- W4381435533
- Semantic Scholar
- 37dc639a4cbd17a891b601e3178ee80340856907
- Cited by
- 96
- References
- 54
- Citations by year
- 2026: 30 · 2025: 25 · 2024: 30 · 2023: 10
- FWCI
- 6.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Funders
- SURFsara (EINF-156)
- Innovation Fund for IT in Research Projects, Utrecht University, The Netherlands
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- active learning (machine learning), systematic review screening, title and abstract screening, classifier comparison, naive Bayes, TF-IDF, Doc2Vec, memory retrieval
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Bayes Theorem, Computer Simulation, Humans, Software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records. https://app.cochat.ai/papers/10.1186/s13643-023-02257-7
- The paper
- Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023). Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records. Systematic Reviews. https://doi.org/10.1186/s13643-023-02257-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.