Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 13Machine Learning and AlgorithmsMeta-analysis and systematic reviewsInformation Retrieval and Search Behavior

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; none of the 32 sources its claims cite is checked for support yet.

Large-scale simulation study of active learning models for systematic reviews

Open access
Teijema, Jelle Jasper, de Bruin, Jonathan, Bagheri, Ayoub, van de Schoot, Rens·2025·International Journal of Data Science and Analytics
DOI 10.1007/s41060-025-00777-0
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite progress in active learning, evaluation remains limited by constraints in simulation size, infrastructure, and dataset availability. This study advocates for large-scale simulations as the gold standard for evaluating active learning models in systematic review screening. Two large-scale simulations, totaling over 29 thousand runs, assessed active learning solutions. The first study evaluated 13 combinations of classification models and feature extraction techniques using high-quality datasets from the SYNERGY dataset. The second expanded this to 92 model combinations with additional classifiers and feature extractors. In every scenario tested, active learning outperformed random screening. The performance gained varied across datasets, models, and screening progression, ranging from considerable to near-flawless results. The findings demonstrate that active learning consistently outperforms random screening in systematic review tasks, offering significant efficiency gains. While the extent of improvement varies depending on the dataset, model choice, and screening stage, the overall advantage is clear. Since model performance differs, active learning systems should remain adaptable to accommodate new classifiers and feature extraction techniques. The publicly available results underscore the importance of open benchmarking to ensure reproducibility and the development of robust, generalizable active learning strategies.

Overview

CoChat Summary

TL;DR

This large-scale simulation study evaluated active learning for systematic review screening across 26 SYNERGY datasets, using 13 common model combinations in study 1 and 92 expanded combinations in study 2. Active learning consistently outperformed random screening, but no single classifier–feature extractor combination was universally best across datasets or screening stages.

Key findings

  • Across all tested scenarios, active learning outperformed random screening; performance gains ranged from marginal to near-flawless depending on dataset, model, and screening stage.
  • No model was universally optimal: out of 26 datasets, 14 different classifier–feature extractor combinations achieved the lowest loss at least once, and the most frequent top performer, “mxbai-embed-large-v1 transformer with Random Forest,” was best in only 7 of 26 cases.
  • For starting performance, the best models identified an average of 27 relevant records within the first 100 screened records, against a theoretical maximum of 51; 44 of 92 models reached the top-performing spot at least once.
  • For last-to-find records, advanced models significantly outperformed simpler models, and Random Forest with mxbai-embed-large-v1 was recommended for identifying difficult final records.

Methods

  • Two simulation studies used 26 prelabeled systematic review datasets from SYNERGY, spanning medicine, psychology, computational sciences, and biology, with dataset sizes from 238 to 48,375 records and relevance densities from <0.25% to >20%.
  • Study 1 evaluated 13 classifier–feature extractor combinations using the ARFI template with one relevant record plus 10 fixed irrelevant records as prior knowledge; study 2 evaluated 92 combinations using the MultiModel template with 5 relevant and 10 irrelevant prior records.
  • Analyses used Normalized Recall Regret (Loss), Work Saved over Sampling (WSS@10%–WSS@100%), starting performance within the first 100 records, last-to-find performance, and Average Pair Distance heatmaps; simulations ran via ASReview on a Kubernetes cluster.

Limitations

  • The paper states that not all possible configurations were covered: samplers and balancers were unexplored, models were treated as feature extractor–classifier combinations rather than separately, and hyperparameter optimization was not performed.
  • The paper states that dataset characteristics such as size or topic showed no significant correlation with model preference, but 26 datasets were insufficient to draw conclusions, and SYNERGY remains 50% “medicine, NOS.”
  • Evident from the text: the study relies on prelabeled retrospective datasets and simulated screening, so real-world reviewer behavior, stopping-rule interactions, and LLM-based classification remain outside the tested scope.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s41060-025-00777-0

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Distribution of fields in simulation studies employing more than four datasets, as found in [53]
Fig. 1 Distribution of fields in simulation studies employing more than four datasets, as found in [53]CC-BY · From the PDF
Fig. 2 Hypothetical example recall curve with a clear indication for clustering records
Fig. 2 Hypothetical example recall curve with a clear indication for clustering recordsCC-BY · From the PDF
Fig. 3 Illustration of the variability in Loss inter-datasets. Datasets are ordered ascending based on their mean Loss values, from best to worst. The plot aims to highlight the dispersion in model performance when applied to different datasets
Fig. 3 Illustration of the variability in Loss inter-datasets. Datasets are ordered ascending based on their mean Loss values, from best to worst. The plot aims to highlight the dispersion in model performance when applied to different datasetsCC-BY · From the PDF
Fig. 4 The intra-dataset variability for each model and dataset combination, represented by Loss. Ordered based on the order of Fig. 3 and mean model loss
Fig. 4 The intra-dataset variability for each model and dataset combination, represented by Loss. Ordered based on the order of Fig. 3 and mean model lossCC-BY · From the PDF
Image 5
CC-BY · From the PDF
Fig. 6 Combined representation of the Average Pair Distance (APD) heatmap ( 3 , 4 ) and recall curves for the dataset Jeyaraman 2020 with the Logistic Regression model and all-mpnet-base-v2 feature extraction. The heatmap uses color coding to indicate pair distances, where the pair distance is defined as the log difference in discovery time between
Fig. 6 Combined representation of the Average Pair Distance (APD) heatmap ( 3 , 4 ) and recall curves for the dataset Jeyaraman 2020 with the Logistic Regression model and all-mpnet-base-v2 feature extraction. The heatmap uses color coding to indicate pair distances, where the pair distance is defined as the log difference in discovery time betweenCC-BY · From the PDF
Image 7
CC-BY · From the PDF
Fig. 8 Plot showing the performance of all active learning models, in Work Saved over Sampling. Also showing the derivative plots only using mxbai or logistic regression
Fig. 8 Plot showing the performance of all active learning models, in Work Saved over Sampling. Also showing the derivative plots only using mxbai or logistic regressionCC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF

Verification

Claims the paper makes

36 Own result · 26 Own interpretation · 20 Not checked yet · 6 Own method · 2 Check pending · 1 No full text of the sources

57 more claims

Create a free account to see all 91 claims and to verify the 29 references still open.

See all claims — free account
  • It may state two separate claims as one. 14
  • Probably not a substantial claim. 16
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Create a free account to see these assumptions.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 63 references

Research works printed but not linked: 2

  1. Molinari, A., Kanoulas, E.: Transferring knowledge between topics in systematic reviews. Intell. Syst. Appl. 16, 200150 (2022)
  2. Xuan, Q., Jiali, L., Yuning, W., et al.: Application of natural language processing in systematic reviews. Chin. J. Evid.-Based Med. 21(6), 715–720 (2021). https://doi.org/10.7507/1672-2531.202012150 · DOI

Cited by

Related

See all

Metadata

Authors
Teijema, J. J., de Bruin, J., Bagheri, A., & van de Schoot, R.
Affiliations
Utrecht University (NL)
Corresponding author
Teijema, Jelle Jasper
ORCID
Venue
International Journal of Data Science and Analytics
Publisher
Springer Science and Business Media LLC
Published
2025-05-02
Volume
20
Issue
6
Pages
5435-5456
ISSN
  • 2364-4168 (ISSN-L)
  • 2364-415X
DOI
10.1007/s41060-025-00777-0
OpenAlex
W4410030482
Semantic Scholar
a1024997e1ec0350279625c7185443a1bd3d0c2d
Cited by
13
References
63
Citations by year
2026: 10 · 2025: 3
FWCI
0.72
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Data Science and Analytics (publishedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
Funders
OCRE
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
active learning (machine learning), systematic review screening, large-scale simulation, classifier comparison, feature extraction, random screening, benchmarking, reproducibility, screening efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.