Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 9

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study

Byrne, Fionn, Hofstee, Laura, Teijema, Jelle, De Bruin, Jonathan, van de Schoot, Rens·2024·Systematic Reviews
DOI 10.1186/s13643-024-02587-0PMC PMC11232241
Open this paper in CoChat — free account

Abstract

Software that employs screening prioritization through active learning (AL) has accelerated the screening process significantly by ranking an unordered set of records by their predicted relevance. However, failing to find a relevant paper might alter the findings of a systematic review, highlighting the importance of identifying elusive papers. The time to discovery (TD) measures how many records are needed to be screened to find a relevant paper, making it a helpful tool for detecting such papers. The main aim of this project was to investigate how the choice of the model and prior knowledge influence the TD values of the hard-to-find relevant papers and their rank orders. A simulation study was conducted, mimicking the screening process on a dataset containing titles, abstracts, and labels used for an already published systematic review. The results demonstrated that AL model choice, and mostly the choice of the feature extractor but not the choice of prior knowledge, significantly influenced the TD values and the rank order of the elusive relevant papers. Future research should examine the characteristics of elusive relevant papers to discover why they might take a long time to be found.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-024-02587-0

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 The time to discovery (TD) values (A) and the rank-orders (B) of all 48 relevant papers across 10 simulations. Each line represents the TD value for a paper across active learning models (grouped by feature extractor). The five hard-to-find relevant papers are highlighted (record IDs contained in the legend)
Fig. 1 The time to discovery (TD) values (A) and the rank-orders (B) of all 48 relevant papers across 10 simulations. Each line represents the TD value for a paper across active learning models (grouped by feature extractor). The five hard-to-find relevant papers are highlighted (record IDs contained in the legend)CC BY · PubMed Central
Fig. 2 Standard deviations (SDs) of the average-record-time to discovery (TD) values for each relevant paper across active learning models (A) and prior knowledge (B). The five hard-to-find relevant papers are highlighted. Each dot represents the average-record-TD value of a paper, and the line corresponds to this value’s SD
Fig. 2 Standard deviations (SDs) of the average-record-time to discovery (TD) values for each relevant paper across active learning models (A) and prior knowledge (B). The five hard-to-find relevant papers are highlighted. Each dot represents the average-record-TD value of a paper, and the line corresponds to this value’s SDCC BY · PubMed Central
Fig. 3 The time to discovery (TD) values (A) and the rank-orders of the TD values (B) of all 48 relevant papers across 48 simulations. Each line represents the TD values of a paper across prior knowledge. The gaps in the lines represent the condition in which a paper is used as prior knowledge. The five hard-to-find relevant papers are highlighted (record IDs listed in the legend)
Fig. 3 The time to discovery (TD) values (A) and the rank-orders of the TD values (B) of all 48 relevant papers across 48 simulations. Each line represents the TD values of a paper across prior knowledge. The gaps in the lines represent the condition in which a paper is used as prior knowledge. The five hard-to-find relevant papers are highlighted (record IDs listed in the legend)CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Research works printed but not linked: 18

  1. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326. https://doi.org/10.1371/journal.pmed.1000326. · DOI · Paper page
  2. Fu J, Lee S. Certainty-based active learning for sampling imbalanced datasets. Neurocomputing. 2013;119:350–8. https://doi.org/10.1016/j.neucom.2013.03.023. · DOI
  3. Gargon E, Gorst SL, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 5th annual update to a systematic review of core outcome sets for research. PLoS ONE. 2019;14(12):e0225980. https://doi.org/10.1371/journal.pone.0225980. · DOI
  4. Le, Q. V., & Mikolov, T. (2014). Distributed representations of sentences and documents (arXiv:1405.4053). arXiv. https://doi.org/10.48550/arXiv.1405.4053. · DOI
  5. Miwa M, Thomas J, O’Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014;51:242–53. https://doi.org/10.1016/j.jbi.2014.06.005. · DOI · Paper page
  6. Nedelcu A, Oerther B, Engel H, Sigle A, Schmucker C, Schoots IG, Eisenblätter M, Benndorf M. A machine learning framework reduces the manual workload for systematic reviews of the diagnostic performance of prostate magnetic resonance imaging. European Urology Open Science. 2023;56:11–4. · DOI
  7. Oud M, Arntz A, Hermens ML, Verhoef R, Kendall T. Specialized psychotherapies for adults with borderline personality disorder: a systematic review and meta-analysis. Aust N Z J Psychiatry. 2018;52(10):949–61. https://doi.org/10.1177/0004867418791257. · DOI · Paper page
  8. Radjenović D, Heričko M, Torkar R, Živkovič A. Software fault prediction metrics: a systematic literature review. Inf Softw Technol. 2013;55(8):1397–418. https://doi.org/10.1016/j.infsof.2013.02.009. · DOI
  9. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: sentence embeddings using Siamese BERT-networks (arXiv:1908.10084). arXiv. https://doi.org/10.48550/arXiv.1908.10084. · DOI · Paper page
  10. Scells, H. (2017, September 1). Reducing workload of systematic review searching and screening processes. Seventh BCS-IRSG Symposium on Future Directions in Information Access. https://doi.org/10.14236/ewic/FDIA2017.2. · DOI
  11. Scherhag J, Burgard T. Performance of semi-automated screening using Rayyan and ASReview: a retrospective analysis of potential work reduction and different stopping rules. Big Data & Research Syntheses 2023: Frankfurt, Germany; 2023.
  12. Subasi, A. (2019). Chapter 4—Feature extraction and dimension reduction. In A. Subasi (Ed.), Practical guide for biomedical signals analysis using machine learning techniques (pp. 193–275). Academic Press. https://doi.org/10.1016/B978-0-12-817444-9.00004-0. · DOI
  13. Teijema, J. J., Seuren, S., Anadria, D., Bagheri, A., & Schoot, R. van de. (2023). Simulation-based active learning for systematic reviews: a systematic review of the literature. PsyArXiv. https://doi.org/10.31234/osf.io/67zmt. · DOI · Paper page
  14. Vlachos A. A stopping criterion for active learning. Comput Speech Lang. 2008;22(3):295–312. https://doi.org/10.1016/j.csl.2007.12.001. · DOI · Paper page
  15. Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19(1):132. https://doi.org/10.1186/s12874-019-0782-0. · DOI · Paper page
  16. Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2022). Neural rankers for effective screening prioritisation in medical systematic review literature search. Proceedings of the 26th Australasian Document Computing Symposium, 1–10. https://doi.org/10.1145/3572960.3572980. · DOI
  17. Yang, E., Lewis, D. D., & Frieder, O. (2021). Heuristic stopping rules for technology-assisted review. Proceedings of the 21st ACM Symposium on Document Engineering, 1–10. https://doi.org/10.1145/3469096.3469873. · DOI
  18. Teijema, J. J., van de Schoot, R., Ferdinands, G., Lombaers, P., de Bruin, J. (2024). Makita—A workflow generator for large-scale and reproducible simulation studies mimicking text labeling. https://doi.org/10.1016/j.simpa.2024.100663. · DOI · Paper page

Cited by

See all 9

Metadata

Authors
Byrne, F., Hofstee, L., Teijema, J., De Bruin, J., & van de Schoot, R.
Venue
Systematic Reviews
Publisher
Springer Science and Business Media LLC
Published
2024-07-08
Volume
13
Issue
1
ISSN
2046-4053
DOI
10.1186/s13643-024-02587-0
PMC
PMC11232241
Cited by
9
References
11
Language
en
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study. https://app.cochat.ai/papers/10.1186/s13643-024-02587-0
The paper
Byrne, F., Hofstee, L., Teijema, J., De Bruin, J., & van de Schoot, R. (2024). Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study. Systematic Reviews. https://doi.org/10.1186/s13643-024-02587-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.