Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 10

The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.

Simulation-based Active Learning for Systematic Reviews: A Scoping Review of Literature

Teijema, Jelle Jasper, Ribeiro, Guilherme, Seuren, Sofie, Anadria, Daniel, Bagheri, Ayoub, van de Schoot, Rens·2023
DOI 10.31234/osf.io/67zmt
Open this paper in CoChat — free account

Abstract

This scoping review investigated the utility of active learning in accelerating the systematic review process. We reviewed literature from 2006 onwards and selected 60 relevant articles out of 1887 screened, while incorporating 238 out of 336 collected datasets. Our analysis of the articles found that, despite some limitations, all 60 articles provided positive recommendations for using active learning to improve the efficiency of the screening phase in systematic reviews. Future research should focus on standardizing metrics, promoting open data, and diversifying models to advance active learning in systematic reviews.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.31234/osf.io/67zmt

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 PRISMA flowchart representing the workflow for the current study, along with the amount of literature found and discarded in each step.
Figure 1 PRISMA flowchart representing the workflow for the current study, along with the amount of literature found and discarded in each step.CC-BY · From the PDF
Figure 2 In this graph for the screening phase of the systematic review, the blue line represents the amount of relevant papers identified at a certain number of papers screening, a metric known as the recall. The dotted yellow line shows the moment the screening was halted, and the black line represents the hypothetical recall for random screening. The x-axis indicates the total dataset of records.
Figure 2 In this graph for the screening phase of the systematic review, the blue line represents the amount of relevant papers identified at a certain number of papers screening, a metric known as the recall. The dotted yellow line shows the moment the screening was halted, and the black line represents the hypothetical recall for random screening. The x-axis indicates the total dataset of records.CC-BY · From the PDF
Figure 3 Recall graph for literature update phase of this systematic review. The x axis shows the total dataset of records.
Figure 3 Recall graph for literature update phase of this systematic review. The x axis shows the total dataset of records.CC-BY · From the PDF
Figure 4 Histogram visualizing the number of papers per year. While the search was started at 2006, no papers were found between 2006 and 2010.
Figure 4 Histogram visualizing the number of papers per year. While the search was started at 2006, no papers were found between 2006 and 2010.CC-BY · From the PDF
Figure 5 Boxplot with the number of datasets used in the included papers.
Figure 5 Boxplot with the number of datasets used in the included papers.CC-BY · From the PDF
Figure 6 Barplot showing the frequency of metrics used in studies.
Figure 6 Barplot showing the frequency of metrics used in studies.CC-BY · From the PDF
Figure 7 Stacked bar chart illustrating the actual availability of reported resources in relation to the frequency of their occurrence. The categories on the x-axis represent the types of resources reported, while the y-axis displays the reported frequency. The differently colored segments of each bar denote the actual availability, highlighting discrepancies between what was reported and the reality of access.
Figure 7 Stacked bar chart illustrating the actual availability of reported resources in relation to the frequency of their occurrence. The categories on the x-axis represent the types of resources reported, while the y-axis displays the reported frequency. The differently colored segments of each bar denote the actual availability, highlighting discrepancies between what was reported and the reality of access.CC-BY · From the PDF
Figure 8 Density plot of the log-transformed number of records in each dataset, with the x-axis representing the number of records and the y-axis indicating the density. This visualization highlights the distribution of dataset sizes within the reviewed studies.
Figure 8 Density plot of the log-transformed number of records in each dataset, with the x-axis representing the number of records and the y-axis indicating the density. This visualization highlights the distribution of dataset sizes within the reviewed studies.CC-BY · From the PDF
Figure 9 Horizontal bar chart depicting the distribution of fields across the datasets under review. The y-axis represents various fields, while the x-axis indicates the relative frequency of each field. This visualization demonstrates the prevalence of different fields within the analyzed datasets.
Figure 9 Horizontal bar chart depicting the distribution of fields across the datasets under review. The y-axis represents various fields, while the x-axis indicates the relative frequency of each field. This visualization demonstrates the prevalence of different fields within the analyzed datasets.CC-BY · From the PDF
Figure 10 Scatterplot depicting the relationship between the number of records and the number of inclusions, with outliers removed for clarity. The x-axis represents the number of records, while the y-axis displays the number of inclusions. The plot highlights the association between these two variables within the context of the analyzed datasets.
Figure 10 Scatterplot depicting the relationship between the number of records and the number of inclusions, with outliers removed for clarity. The x-axis represents the number of records, while the y-axis displays the number of inclusions. The plot highlights the association between these two variables within the context of the analyzed datasets.CC-BY · From the PDF
Figure 11 Horizontal bar chart showcasing the frequency of machine learning model choices employed in the studies. The y-axis represents various machine learning models, while the x-axis indicates their frequency of usage. This visualization highlights the popularity of different models within the context of the analyzed studies.
Figure 11 Horizontal bar chart showcasing the frequency of machine learning model choices employed in the studies. The y-axis represents various machine learning models, while the x-axis indicates their frequency of usage. This visualization highlights the popularity of different models within the context of the analyzed studies.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 44

  1. I. J. Marshall and B. C. Wallace. "Toward systematic review automation: a practical guide to using machine learning tools in research synthesis". In: Syst Rev 8.1 (2019), p. 163. issn: 2046-4053 (Electronic) 2046-4053 (Linking). doi: 10.1186/s13643-019-1074-9. · DOI · Paper page
  2. Julian PT Higgins et al. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons, 2019.
  3. Gaelen P. Adam, Byron C. Wallace, and Thomas A. Trikalinos. "Semi-automated Tools for Systematic Searches". In: Methods in Molecular Biology. Springer US, Sept. 2021, pp. 17-40. doi: 10.1007/978-1-0716-1566-9_2. url: https://doi.org/10.1007/978-1-0716-1566-9_2. · DOI · Paper page
  4. Gerit Wagner, Roman Lukyanenko, and Guy Paré. "Artificial intelligence and the conduct of literature reviews". In: Journal of Information Technology 37.2 (2022), pp. 209-226.
  5. Lucy Lu Wang and Kyle Lo. "Text mining approaches for dealing with the rapidly expanding literature on COVID-19". In: Briefings in Bioinformatics 22.2 (2021), pp. 781-799.
  6. Q. Xuan et al. "Application of natural language processing in systematic reviews". In: Chinese Journal of Evidence-Based Medicine 21(6) (2021), 715-720. doi: 10.7507/1672-2531.202012150. url: https://doi. org/10.7507/1672-2531.202012150. · DOI
  7. Lauke Stoel, Gijs Mourits, and Rens van de Schoot. Procedure and Results for the Initial Selection of Software for Systematically Screening Large Amounts of Textual Data Implementing Active Learning. 2023. doi: 10. 17605/OSF.IO/G3NKZ. url: osf.io/g3nkz. · DOI
  8. Makoto Miwa et al. "Reducing systematic review workload through certainty-based screening". In: Journal of Biomedical Informatics 51 (2014), pp. 242-253. issn: 15320464. doi: 10.1016/j.jbi.2014.06.005. · DOI · Paper page
  9. James Thomas et al. "Living systematic reviews: 2. Combining human and machine effort". In: Journal of clinical epidemiology 91 (2017), pp. 31-37. issn: 0895-4356.
  10. Sofie A G E van den Brand, Rens van de Schoot, and J J Teijema. A Systematic Review on Studies Evaluating the Performance of Active Learning Compared to Human Reading for Systematic Review Data. July 2021. doi: 10.17605/OSF.IO/T9HGM. url: osf.io/t9hgm. · DOI
  11. Sathish Muthu. "The efficiency of machine learning-assisted platform for article screening in systematic reviews in orthopaedics". In: International Orthopaedics (2022), pp. 1-6.
  12. Gerbrich Ferdinands. "AI-Assisted systematic reviewing: selecting studies to compare Bayesian versus Frequen- tist SEM for small sample sizes". In: Multivariate Behavioral Research 56.1 (2021), pp. 153-154.
  13. B. C. Wallace et al. "Modeling annotation time to reduce workload in comparative effectiveness reviews". In: pp. 28-35. doi: 10.1145/1882992.1882999. · DOI · Paper page
  14. B. C. Wallace et al. "Active learning for biomedical citation screening". In: pp. 173-181. doi: 10 . 1145 / 1835804.1835829. · DOI · Paper page
  15. B. C. Wallace et al. "Who should label what? instance allocation in multiple expert active learning". In: pp. 176-187. doi: 10.1137/1.9781611972818.16. · DOI
  16. Byron C. Wallace et al. "Deploying an interactive machine learning system in an evidence-based practice center: abstrackr". In: the 2nd ACM SIGHIT symposium. Miami, Florida, USA: ACM Press, p. 819. isbn: 978-1-4503-0781-9. doi: 10.1145/2110363.2110464. · DOI · Paper page
  17. K. H. Ambert et al. "Virk: An active learning-based system for bootstrapping knowledge base development in the neurosciences". In: Frontiers in Neuroinformatics 7.DEC (2013). doi: 10.3389/fninf.2013.00038. · DOI · Paper page
  18. Gordon V. Cormack and Maura R. Grossman. "Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review". In: arXiv:1504.06868 [cs] (2015). · arXiv
  19. P. Timsina, O. El-Gayar, and J. Liu. "Active learning for the automation of medical systematic review creation". In.
  20. Kazuma Hashimoto et al. "Topic detection using paragraph vectors to support active learning in systematic reviews". In: JOURNAL OF BIOMEDICAL INFORMATICS 62 (2016), pp. 59-65. issn: 1532-0464. doi: 10.1016/j.jbi.2016.06.001. · DOI
  21. Gordon V. Cormack and Maura R. Grossman. "Scalability of Continuous Active Learning for Reliable High- Recall Text Classification". In: CIKM'16: ACM Conference on Information and Knowledge Management. Indianapolis Indiana USA: ACM, pp. 1039-1048. isbn: 978-1-4503-4073-1. doi: 10.1145/2983323.2983776. · DOI
  22. G. V. Cormack and M. R. Grossman. "Technology-assisted review in empirical medicine: Waterloo participation in CLEF eHealth 2017". In: vol. 1866.
  23. Z. Yu and T. Menzies. "Data balancing for technologically assisted reviews: Undersampling or reweighting". In: vol. 1866.
  24. Rasmus Ros, Elizabeth Bjarnason, and Per Runeson. "A machine learning approach for semi-automated search and selection in literature studies". In: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering. 2017, pp. 118-127.
  25. C. Norman, M. Leeflang, and A. Névéol. "LIMSI@CLEF eHealth 2018 Task 2: Technology assisted reviews by stacking active and static learning". In: vol. 2125.
  26. Jun Liu, Prem Timsina, and Omar El-Gayar. "A comparative analysis of semi-supervised learning: The case of article selection for medical systematic reviews". In: INFORMATION SYSTEMS FRONTIERS 20.2, SI (2018), pp. 195-207. issn: 1387-3326. doi: 10.1007/s10796-016-9724-0. · DOI
  27. G. V. Cormack and M. R. Grossman. "Technology-assisted review in empirical medicine: Waterloo participation in CLEF eHealth 2018". In: vol. 2125.
  28. G. M. Di Nunzio. "Finding all the needles in a haystack: A System to estimate the costs of e-discovery and systematic reviews". In: vol. 2167, p. 106.
  29. E. Kanoulas et al. "CLEF 2019 technology assisted reviews in empirical medicine overview". In: vol. 2380.
  30. Christopher R. Norman et al. "Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy". In: SYSTEMATIC REVIEWS 8.1 (2019). doi: 10.1186/s13643-019-1162-x. · DOI
  31. A. Carvallo and D. Parra. "Comparing word embeddings for document screening based on active learning". In: vol. 2414, pp. 100-107.
  32. A. Varghese et al. "Active learning in automated text classification: a case study exploring bias in predicted model performance metrics". In: Environment Systems and Decisions 39.3 (2019), pp. 269-280. doi: 10.1007/ s10669-019-09717-3. · DOI
  33. N. Halfpenny et al. "PNS335: Using Machine Learning for Efficiency Improvements in Systematic Literature Reviews of Clinical Efficacy and Safety". In: Value in Health 22.(Halfpenny N.; Eaton J.) Pharmerit Interna- tional, York, United Kingdom (2019), S821. issn: 1524-4733. doi: 10.1016/j.jval.2019.09.2235. · DOI
  34. J. Zou and E. Kanoulas. "Towards Question-based High-recall Information Retrieval". In: ACM Transactions on Information Systems 38.3 (2020). doi: 10.1145/3388640. · DOI
  35. H. Zhang et al. "Evaluating sentence-level relevance feedback for high-recall information retrieval". In: Infor- mation Retrieval Journal 23.1 (2020). doi: 10.1007/s10791-019-09361-0. · DOI
  36. Tomohide Yamada et al. "Deep Neural Network for Reducing the Screening Workload in Systematic Reviews for Clinical Guidelines: Algorithm Validation Study". In: Journal of Medical Internet Research 22.12 (2020), e22422. issn: 1438-8871. doi: 10.2196/22422. · DOI
  37. Andres Carvallo et al. "Automatic document screening of medical literature using word and text embeddings in an active learning setting". In: SCIENTOMETRICS 125.3 (2020), pp. 3047-3084. issn: 0138-9130. doi: 10.1007/s11192-020-03648-6. · DOI
  38. D. Li, P. Zafeiriadis, and E. Kanoulas. "APS: An Active PubMed Search System for Technology Assisted Reviews". In: pp. 2137-2140. doi: 10.1145/3397271.3401401. · DOI
  39. Rebecca Elmore et al. "Risk and Protective Factors in the COVID-19 Pandemic: A Rapid Evidence Map". In: FRONTIERS IN PUBLIC HEALTH 8 (2020). doi: 10.3389/fpubh.2020.582205. · DOI
  40. Zhichao Li et al. "Semi-Supervised Text Classification Framework: An Overview of Dengue Landscape Factors and Satellite Earth Observation". In: INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 17.12 (2020). doi: 10.3390/ijerph17124509. · DOI
  41. Niamh Carey, Marie Harte, and Laura McCullagh. "The Use Of A Text-Mining Screening Tool For Systematic Review Of Treatments For Relapsed/Refractory Diffuse Large B-Cell Lymphoma". In: International Journal of Technology Assessment in Health Care 37.S1 (2021), pp. 2-2.
  42. Jingwen Hou et al. "Extreme Systematic Reviews: A Large Literature Screening Dataset to Support Environ- mental Policymaking". In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022, pp. 4029-4033.
  43. Guilherme Dal Bianco, Denio Duarte, and Marcos André Gonçalves. "Reducing the user labeling effort in effective high recall tasks by fine-tuning active learning". In: Journal of Intelligent Information Systems (2023), pp. 1-20.
  44. B. K. Olorisade, P. Brereton, and P. Andras. "Reproducibility of studies on text mining for citation screening in systematic reviews: Evaluation and checklist". In: J Biomed Inform 73 (2017). Olorisade, Babatunde Kazeem Brereton, Pearl Andras, Peter eng 2017/07/18 J Biomed Inform. 2017 Sep;73:1-13. doi: 10.1016/j.jbi.2017.07.010. Epub 2017 Jul 12., pp. 1-13. issn: 1532-0480 (Electronic) 1532-0464 (Linking). doi: 10.1016/j.jbi.2017. 07.010. · DOI · Paper page

Cited by

See all 10

Metadata

Authors
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R.
Publisher
Center for Open Science
Published
2023-06-29
DOI
10.31234/osf.io/67zmt
Cited by
10
References
23
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode (unspecified)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.