The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study
Abstract
Software that employs screening prioritization through active learning (AL) has accelerated the screening process significantly by ranking an unordered set of records by their predicted relevance. However, failing to find a relevant paper might alter the findings of a systematic review, highlighting the importance of identifying elusive papers. The time to discovery (TD) measures how many records are needed to be screened to find a relevant paper, making it a helpful tool for detecting such papers. The main aim of this project was to investigate how the choice of the model and prior knowledge influence the TD values of the hard-to-find relevant papers and their rank orders. A simulation study was conducted, mimicking the screening process on a dataset containing titles, abstracts, and labels used for an already published systematic review. The results demonstrated that AL model choice, and mostly the choice of the feature extractor but not the choice of prior knowledge, significantly influenced the TD values and the rank order of the elusive relevant papers. Future research should examine the characteristics of elusive relevant papers to discover why they might take a long time to be found.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Based on the findings of [7], we set the prior knowledge to a level that ensures stability, using a number of records that will minimize early-cycle variability while maintaining a realistic screening scenario.”
From Large-scale simulation study of active learning models for systematic reviews (2025) · cites this paper as [7]
In the citing paper
“While useful for evaluating performance at a fixed recall level, WSS@95% provides no insight into how a model performs in scenarios such as quickly retrieving relevant records after a cold start or identifying the last-to-find records [7,19,22].”
From Large-scale simulation study of active learning models for systematic reviews (2025) · cites this paper as [7]
In the citing paper
“For last-to-find records [7,19,22], advanced models significantly outperform simpler models due to their ability to detect contextual and semantic relationships.”
From Large-scale simulation study of active learning models for systematic reviews (2025) · cites this paper as [7]
In the citing paper
“Additionally, it can help detect “odd” relevant records28 that appear isolated from most other records but warrant closer inspection29 or to identify noisy labels.35”
From ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts (2025) · cites this paper as [28]
In the citing paper
“So, ASReview has demonstrated its ability to speed up systematic reviews.26,27,28,29,30,31,32,33,34,35,36”
From ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts (2025) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 1×
“Screening prioritization through active learning (AL) has enabled the screening process to be sped up significantly by ranking an unordered set of records by their predicted relevance [17, 23, 25, 29, 32].”
BackgroundSelf-citationBorah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 1×
“Historically, this issue has been noted by studies such as Bastian et al. [3], and Borah et al. [5], highlighting its long-standing nature.”
BackgroundReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 1×
“It enables the user to theoretically find relevant papers by screening only a fraction of the most likely relevant records [9].”
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146 · cited 2×
“Let us assume that the screener uses a (naïve) stopping rule of 100 irrelevant papers in a row (more on stopping rules: [4, 8, 14, 30, 33] and the screener indeed is presented 100 irrelevant records in a row and decides to stop screening.”
Self-citationStatistical stopping criteria for automated screening in systematic reviews
Callaghan, M. W., & Müller-Hansen, F. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01521-4 · Cited by 121 · cited 1×
“Let us assume that the screener uses a (naïve) stopping rule of 100 irrelevant papers in a row (more on stopping rules: [4, 8, 14, 30, 33] and the screener indeed is presented 100 irrelevant records in a row and decides to stop screening.”
BackgroundFerdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02257-7 · Cited by 96 · cited 2×
“Previous research on TD has examined the ATD (or average-simulation TD in the context of simulation studies) across different models and datasets [12].”
BackgroundSelf-citationASReview LAB - A tool for AI-assisted systematic reviews
developers, A. L. (2026) · Zenodo · doi:10.5281/zenodo.3345592 · Cited by 59
Active learning for screening prioritization in systematic reviews - A simulation study
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020) · doi:10.31219/osf.io/w6qbg · Cited by 51
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., et al. (2023) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2023.1178181 · Cited by 36
Self-citationSYNERGY - Open machine learning dataset on study selection in systematic reviews
De Bruin, J., Ma, Y., Ferdinands, G., Teijema, J., & Van de Schoot, R. (2023) · DataverseNL · doi:10.34894/he6naq · Cited by 20
Self-citationBrouwer, M., Hofstee, L., van den Brand, S. A. G. E., Teijema, J. J., Ferdinands, G., de Boer, J., Weijdema, F., Kramer, B., Wiers, R., Bockting, C., et al. (2022) · doi:10.31234/osf.io/j6nqz · Cited by 11
Self-citation
Research works printed but not linked: 18
- Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326. https://doi.org/10.1371/journal.pmed.1000326. · DOI · Paper page
- Fu J, Lee S. Certainty-based active learning for sampling imbalanced datasets. Neurocomputing. 2013;119:350–8. https://doi.org/10.1016/j.neucom.2013.03.023. · DOI
- Gargon E, Gorst SL, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLoS ONE. 2019;14(12):e0225980. https://doi.org/10.1371/journal.pone.0225980. · DOI
- Le, Q. V., & Mikolov, T. (2014). Distributed representations of sentences and documents (arXiv:1405.4053). arXiv. https://doi.org/10.48550/arXiv.1405.4053. · DOI
- Miwa M, Thomas J, O’Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242–53. https://doi.org/10.1016/j.jbi.2014.06.005. · DOI · Paper page
- Nedelcu A, Oerther B, Engel H, Sigle A, Schmucker C, Schoots IG, Eisenblätter M, Benndorf M. A machine learning framework reduces the manual workload for systematic reviews of the diagnostic performance of prostate magnetic resonance imaging. European Urology Open Science. 2023;56:11–4. · DOI
- Oud M, Arntz A, Hermens ML, Verhoef R, Kendall T. Specialized psychotherapies for adults with borderline personality disorder: a systematic review and meta-analysis. Aust N Z J Psychiatry. 2018;52(10):949–61. https://doi.org/10.1177/0004867418791257. · DOI · Paper page
- Radjenović D, Heričko M, Torkar R, Živkovič A. Software fault prediction metrics: a systematic literature review. Inf Softw Technol. 2013;55(8):1397–418. https://doi.org/10.1016/j.infsof.2013.02.009. · DOI
- Reimers, N., & Gurevych, I. (2019). Sentence-BERT: sentence embeddings using Siamese BERT-networks (arXiv:1908.10084). arXiv. https://doi.org/10.48550/arXiv.1908.10084. · DOI · Paper page
- Scells, H. (2017, September 1). Reducing workload of systematic review searching and screening processes. Seventh BCS-IRSG Symposium on Future Directions in Information Access. https://doi.org/10.14236/ewic/FDIA2017.2. · DOI
- Scherhag J, Burgard T. Performance of semi-automated screening using Rayyan and ASReview: a retrospective analysis of potential work reduction and different stopping rules. Big Data & Research Syntheses 2023: Frankfurt, Germany; 2023.
- Subasi, A. (2019). Chapter 4—Feature extraction and dimension reduction. In A. Subasi (Ed.), Practical guide for biomedical signals analysis using machine learning techniques (pp. 193–275). Academic Press. https://doi.org/10.1016/B978-0-12-817444-9.00004-0. · DOI
- Teijema, J. J., Seuren, S., Anadria, D., Bagheri, A., & Schoot, R. van de. (2023). Simulation-based active learning for systematic reviews: a systematic review of the literature. PsyArXiv. https://doi.org/10.31234/osf.io/67zmt. · DOI · Paper page
- Vlachos A. A stopping criterion for active learning. Comput Speech Lang. 2008;22(3):295–312. https://doi.org/10.1016/j.csl.2007.12.001. · DOI · Paper page
- Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19(1):132. https://doi.org/10.1186/s12874-019-0782-0. · DOI · Paper page
- Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2022). Neural rankers for effective screening prioritisation in medical systematic review literature search. Proceedings of the 26th Australasian Document Computing Symposium, 1–10. https://doi.org/10.1145/3572960.3572980. · DOI
- Yang, E., Lewis, D. D., & Frieder, O. (2021). Heuristic stopping rules for technology-assisted review. Proceedings of the 21st ACM Symposium on Document Engineering, 1–10. https://doi.org/10.1145/3469096.3469873. · DOI
- Teijema, J. J., van de Schoot, R., Ferdinands, G., Lombaers, P., de Bruin, J. (2024). Makita—A workflow generator for large-scale and reproducible simulation studies mimicking text labeling. https://doi.org/10.1016/j.simpa.2024.100663. · DOI · Paper page
Cited by
ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts
de Bruin, J., Lombaers, P., Kaandorp, C., Teijema, J., van der Kuil, T., Yazan, B., Dong, A., & van de Schoot, R. (2025) · Patterns · doi:10.1016/j.patter.2025.101318 · Cited by 27
Self-citationMachine learning to optimize literature screening in medical guideline development
Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., de Bruin, J., van den Brand, S., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02590-5 · Cited by 21
Self-citationLarge-scale simulation study of active learning models for systematic reviews
Teijema, J. J., de Bruin, J., Bagheri, A., & van de Schoot, R. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00777-0 · Cited by 13
Self-citationPitre, T., Zeraatkar, D., Granton, J., Busse, J., Rochwerg, B., & Guyatt, G. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112514 · Cited by 2
Metadata
- Authors
- Byrne, F., Hofstee, L., Teijema, J., De Bruin, J., & van de Schoot, R.
- Venue
- Systematic Reviews
- Published
- 2024-07-08
- Volume
- 13
- Issue
- 1
- ISSN
- 2046-4053
- PMC
- PMC11232241
- Cited by
- 9
- References
- 11
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
