CoChat

Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records

Open access
Ferdinands, Gerbrich, Schram, Raoul, de Bruin, Jonathan, Bagheri, Ayoub, Oberski, Daniel L., Tummers, Lars, Teijema, Jelle Jasper, van de Schoot, Rens · 2023 · Systematic Reviews

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 28 of the 38 sources its claims cite were checked for support.

Abstract

BACKGROUND: Conducting a systematic review demands a significant amount of effort in screening titles and abstracts. To accelerate this process, various tools that utilize active learning have been proposed. These tools allow the reviewer to interact with machine learning software to identify relevant publications as early as possible. The goal of this study is to gain a comprehensive understanding of active learning models for reducing the workload in systematic reviews through a simulation study. METHODS: The simulation study mimics the process of a human reviewer screening records while interacting with an active learning model. Different active learning models were compared based on four classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two feature extraction strategies (TF-IDF and doc2vec). The performance of the models was compared for six systematic review datasets from different research areas. The evaluation of the models was based on the Work Saved over Sampling (WSS) and recall. Additionally, this study introduces two new statistics, Time to Discovery (TD) and Average Time to Discovery (ATD). RESULTS: The models reduce the number of publications needed to screen by 91.7 to 63.9% while still finding 95% of all relevant records (WSS@95). Recall of the models was defined as the proportion of relevant records found after screening 10% of of all records and ranges from 53.6 to 99.8%. The ATD values range from 1.4% till 11.7%, which indicate the average proportion of labeling decisions the researcher needs to make to detect a relevant record. The ATD values display a similar ranking across the simulations as the recall and WSS values. CONCLUSIONS: Active learning models for screening prioritization demonstrate significant potential for reducing the workload in systematic reviews. The Naive Bayes + TF-IDF model yielded the best results overall. The Average Time to Discovery (ATD) measures performance of active learning models throughout the entire screening process without the need for an arbitrary cut-off point. This makes the ATD a promising metric for comparing the performance of different models across different datasets.

Overview

CoChat Summary

TL;DR

This simulation study compared seven active-learning models across six systematic review datasets and introduced Average Time to Discovery (ATD) as a screening-prioritization metric. It found substantial workload reductions, with NB + TF-IDF generally performing best and ATD ranking models similarly to recall and WSS@95.

Key findings

  • Across models and datasets, WSS@95 ranged from 63.9% to 91.7%, meaning models reduced the number of publications needed to screen while still finding 95% of relevant records.
  • Recall after screening 10% of records (RRF@10) ranged from 53.6% to 99.8%; ATD values ranged from 1.4% to 11.7%, indicating the average proportion of labeling decisions needed to detect a relevant record.
  • NB + TF-IDF yielded the best results overall: it ranked first in ACE, Virus, and Wilson, shared first in PTSD and Software, and second in Nudging. RF + TF-IDF never performed best on any measure across datasets.
  • ATD values showed a similar ranking across simulations as recall and WSS values, and all models exceeded random screening.

Methods

  • Simulation study mimicking human title/abstract screening with active learning; seven models combined four classifiers (naive Bayes, logistic regression, support vector machines, random forest) with two feature-extraction strategies (TF-IDF, doc2vec), excluding NB + D2V because multinomial NB requires positive feature values.
  • Six labeled systematic review datasets from medicine, computer science, and social science were used, yielding 42 simulations. Each simulation started with one relevant and one irrelevant record, retrained after each label, ended after all records were labeled, and was repeated 15 times with random initial training sets held constant within dataset.
  • Hyperparameters were optimized via Tree of Parzen Estimators; simulations used ASReview 0.9.3 and analyses used R 3.6.1. Performance metrics were WSS@95, RRF@10, Time to Discovery (TD), and Average Time to Discovery (ATD), with recall curves, means with standard deviations, medians with MAD, and 95% confidence intervals.

Limitations

  • Stated: hyperparameters were optimized for each model-dataset combination, so results reflect maximum model performance and generalizability to datasets without optimized hyperparameters remains unknown; the sample size of six datasets does not allow investigating how data characteristics relate to model performance.
  • Stated: runtime varied widely across models, with implications for practical use because an efficient model should keep up with reviewer decision speed; ATD is not set against a baseline, and more research is needed on its statistical bias, outliers, sparse data, sensitivity to dataset characteristics, and construct/face validity.
  • Evident: the study is a simulation using existing labels rather than real reviewer interaction, and the six datasets are a convenience sample; D2V versus TF-IDF results were inconclusive.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-023-02257-7

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Recall curves of all seven models for a the Nudging, b PTSD, c Software, d ACE, (e) Virus, and (f) Wilson dataset. The figure shows the recall as a function of the proportion of publications screened. The x-axis is cut off at 40% since at this point in screening all models had already reached 95% recall. The dashed horizontal lines indicate the RRF@10 values, the dashed vertical lines the WSS@95 values. The dashed black diagonal line corresponds to the expected recall curve when publications are screened in a random order
Fig. 1 Recall curves of all seven models for a the Nudging, b PTSD, c Software, d ACE, (e) Virus, and (f) Wilson dataset. The figure shows the recall as a function of the proportion of publications screened. The x-axis is cut off at 40% since at this point in screening all models had already reached 95% recall. The dashed horizontal lines indicate the RRF@10 values, the dashed vertical lines the WSS@95 values. The dashed black diagonal line corresponds to the expected recall curve when publications are screened in a random orderCC BY · PubMed Central
Fig. 2 Performance for all datasets, on average across seven models with a 95% confidence interval. Performance is expressed by three different metrics: ATD, WSS@95, and RRF@10. All metrics are measured on a scale from 0 to 100. For the ATD holds that the lower the value, the better the performance. For the WSS and RRF holds that the higher the value, the better the performance
Fig. 2 Performance for all datasets, on average across seven models with a 95% confidence interval. Performance is expressed by three different metrics: ATD, WSS@95, and RRF@10. All metrics are measured on a scale from 0 to 100. For the ATD holds that the lower the value, the better the performance. For the WSS and RRF holds that the higher the value, the better the performanceCC BY · PubMed Central
Algorithm 1 Pseudo code demonstrating the active learning cycle for screening prioritization. The model incrementally improves its predictions on the remaining unlabeled records. Relevant publications are identified as early in the process as possible
Algorithm 1 Pseudo code demonstrating the active learning cycle for screening prioritization. The model incrementally improves its predictions on the remaining unlabeled records. Relevant publications are identified as early in the process as possibleCC BY · PubMed Central
Algorithm 2 Average Time to Discovery (ATD). The Time to Discover (TD) a given relevant record i for a given trial k is given by the number of records needed to screen to detect i. Averaged over all trials, this number is the TDi¯, also called the average-record-TD. The Average Time to Discovery (ATD) is then given by taking the average of all average-record-TD’s and expressed as a fraction of the total number of records needed to screen
Algorithm 2 Average Time to Discovery (ATD). The Time to Discover (TD) a given relevant record i for a given trial k is given by the number of records needed to screen to detect i. Averaged over all trials, this number is the TDi¯, also called the average-record-TD. The Average Time to Discovery (ATD) is then given by taking the average of all average-record-TD’s and expressed as a fraction of the total number of records needed to screenCC BY · PubMed Central
Algorithm 3 Pseudo code for balancing a labeled dataset by Dynamic Resampling. Hyperparameters are a, that governs the weight of the 1’s; α, that governs the scaling the weight of the 1’s, as a function of the ratio of ones to zeros; b, governs how strongly we want to sample depending on the total number of samples; β, governs the scaling of the weight of the zeros depending on the number of samples. Values for these hyperparameters can be found on the GitHub repository of this paper [37]
Algorithm 3 Pseudo code for balancing a labeled dataset by Dynamic Resampling. Hyperparameters are a, that governs the weight of the 1’s; α, that governs the scaling the weight of the 1’s, as a function of the ratio of ones to zeros; b, governs how strongly we want to sample depending on the total number of samples; β, governs the scaling of the weight of the zeros depending on the number of samples. Values for these hyperparameters can be found on the GitHub repository of this paper [37]CC BY · PubMed Central

Verification

Claims the paper makes

15 Valid · 2 Not valid · 3 Not checked · 70 Own results and interpretations

50 more claims · 1 with their cited sources checked

Create a free account to see all 90 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 24
  • It may state two separate claims as one. 16
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 59 references

Research works printed but not linked: 5

  1. Carvallo A, Parra D. Comparing word embeddings for document screening based on active learning. In: BIRNDL@ SIGIR. 2019. p. 100–107.
  2. Le Q, Mikolov T. Distributed representations of sentences and documents. In: International conference on machine learning. Beijing: PMLR; 2014. p. 1188–1196.
  3. Bergstra J, Yamins D, Cox D. Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: International Conference on Machine Learning. 2013. p. 115–123. http://proceedings.mlr.press/v28/bergstra13.html. Accessed Mar 2020.
  4. Mitchell TM. Does Machine Learning Really Work? AI Mag. 1997;18(3):11.
  5. Kanoulas E, Li D, Azzopardi L, Spijker R. CLEF 2019 technology assisted reviews in empirical medicine overview. In: CEUR workshop proceedings, vol. 2380. Lugano: CEU; 2019.

Cited by

See all 96

Metadata

Authors
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R.
Affiliations
Utrecht University (NL)
Corresponding author
Ferdinands, Gerbrich
Venue
Systematic Reviews
Published
2023-06-20
Volume
12
Issue
1
Pages
100-100
ISSN
2046-4053 (ISSN-L)
OpenAlex
W4381435533
Cited by
96
References
54
Citations by year
2026: 30 · 2025: 25 · 2024: 30 · 2023: 10
FWCI
6.70
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
Funders
  • SURFsara (EINF-156)
  • Innovation Fund for IT in Research Projects, Utrecht University, The Netherlands
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
active learning (machine learning), systematic review screening, title and abstract screening, classifier comparison, naive Bayes, TF-IDF, Doc2Vec, memory retrieval
MeSH terms
Machine Learning, Systematic Reviews as Topic, Bayes Theorem, Computer Simulation, Humans, Software

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records. https://app.cochat.ai/papers/10.1186/s13643-023-02257-7
The paper
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023). Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records. Systematic Reviews. https://doi.org/10.1186/s13643-023-02257-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.