Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 20

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews

Lombaers, Peter, de Bruin, Jonathan, van de Schoot, Rens·2024·Applied Sciences
DOI 10.3390/app14093842
Open this paper in CoChat — free account

Abstract

In the screening phase of a systematic review, screening prioritization via active learning effectively reduces the workload. However, the PRISMA guidelines are not sufficient for reporting the screening phase in a reproducible manner. Text screening with active learning is an iterative process, but the labeling decisions and the training of the active learning model can happen independently of each other in time. Therefore, it is not trivial to store the data from both events so that one can still know which iteration of the model was used for each labeling decision. Moreover, many iterations of the active learning model will be trained throughout the screening process, producing an enormous amount of data (think of many gigabytes or even terabytes of data), and machine learning models are continually becoming larger. This article clarifies the steps in an active learning-aided screening process and what data is produced at every step. We consider what reproducibility means in this context and we show that there is tension between the desire to be reproducible and the amount of data that is stored. Finally, we present the RDAL Checklist (Reproducibility and Data storage for Active Learning-Aided Systematic Reviews Checklist), which helps users and creators of active learning software make their screening process reproducible.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/app14093842

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The human-in-theloop (HITL) active learning cycle with two types of output: on the top right, the trained model, which is the main output in the classical active learning; and at the bottom left, all relevant papers, which is the main output of researcher-in-theloop (RITL) active learning. Figure 1. The human-in-the-loop (HITL) active learning cycle with two types of output: on the top right, the trained model, which is the main output in the classical active learning; and at the bottom left, all relevant papers, which is the main output of researcher-in-the-loop (RITL) active learning.
Figure 1 The human-in-theloop (HITL) active learning cycle with two types of output: on the top right, the trained model, which is the main output in the classical active learning; and at the bottom left, all relevant papers, which is the main output of researcher-in-theloop (RITL) active learning. Figure 1. The human-in-the-loop (HITL) active learning cycle with two types of output: on the top right, the trained model, which is the main output in the classical active learning; and at the bottom left, all relevant papers, which is the main output of researcher-in-the-loop (RITL) active learning.CC-BY · From the PDF
Figure 2 Main steps in a systematic review using active learning. Figure 2. Main steps in a systematic review using active learning.
Figure 2 Main steps in a systematic review using active learning. Figure 2. Main steps in a systematic review using active learning.CC-BY · From the PDF
Figure 3 The two cycles of labeling and training. Figure 3. The two cycles of labeling and training.
Figure 3 The two cycles of labeling and training. Figure 3. The two cycles of labeling and training.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 27

  1. Runje, 2024, RCE (rationale–cogency–extent) criterion unravels features affecting citation impact of top-ranked systematic literature reviews: Leaving the impression…is all you need, Scientometrics, 129, 1891, 10.1007/s11192-024-04935-2 · DOI
  2. Grant, 2009, A typology of reviews: An analysis of 14 review types and associated methodologies, Health Inf. Libr. J., 26, 91, 10.1111/j.1471-1842.2009.00848.x · DOI · Paper page
  3. Moher, 2009, Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement, Ann. Intern. Med., 151, 264, 10.7326/0003-4819-151-4-200908180-00135 · DOI · Paper page
  4. Higgins, J.P.T., and Green, S. (2008). Cochrane Handbook for Systematic Reviews of Interventions, John Wiley & Sons, Ltd. · DOI · Paper page
  5. Marshall, 2019, Toward systematic review automation: A practical guide to using machine learning tools in research synthesis, Syst. Rev., 8, 163, 10.1186/s13643-019-1074-9 · DOI · Paper page
  6. Cohen, 2009, Cross-topic learning for work prioritization in systematic review creation and update, J. Am. Med. Informatics Assoc., 16, 690, 10.1197/jamia.M3162 · DOI · Paper page
  7. Thomas, 2017, Living systematic reviews: 2. Combining human and machine effort, J. Clin. Epidemiology, 91, 31, 10.1016/j.jclinepi.2017.08.011 · DOI · Paper page
  8. Settles, B. (2012). Active Learning, Springer. Synthesis Lectures on Artificial Intelligence and Machine Learning. · DOI
  9. Wallace, B.C., Small, K., Brodley, C.E., Lau, J., and Trikalinos, T.A. (2012, January 28). Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. Proceedings of the ACM International Health Informatics Symposium (IHI), New York, NY, USA. · DOI · Paper page
  10. Cheng, 2018, Using machine learning to advance synthesis and use of conservation and environmental evidence, Conserv. Biol., 32, 762, 10.1111/cobi.13117 · DOI · Paper page
  11. Ferdinands, 2021, AI-Assisted Systematic Reviewing: Selecting Studies to Compare Bayesian Versus Frequentist SEM for Small Sample Sizes, Multivar. Behav. Res., 56, 153, 10.1080/00273171.2020.1853501 · DOI
  12. Marshall, I., Kuiper, J., Banner, E., and Wallace, B.C. (August, January 30). Automating Biomedical Evidence Synthesis: Robot Reviewer. Proceedings of the ACL 2017, System Demonstrations, Vancouver, BC, Canada. · DOI · Paper page
  13. Fu, 2013, Certainty-based active learning for sampling imbalanced datasets, Neurocomputing, 119, 350, 10.1016/j.neucom.2013.03.023 · DOI
  14. Groen, 2022, A systematic review on the use of explainability in deep learning systems for computer aided diagnosis in radiology: Limited use of explainable AI?, Eur. J. Radiol., 157, 110592, 10.1016/j.ejrad.2022.110592 · DOI · Paper page
  15. Villalobos, P., Sevilla, J., Besiroglu, T., Heim, L., Ho, A., and Hobbhahn, M. (2022). Machine Learning Model Sizes and the Parameter Gap. arXiv.
  16. Olorisade, 2017, Reproducibility of studies on text mining for citation screening in systematic reviews: Evaluation and checklist, J. Biomed. Informatics, 73, 1, 10.1016/j.jbi.2017.07.010 · DOI · Paper page
  17. ASReview LAB Developers (2024). AS Review LAB—A Tool for AI-Assisted Systematic Reviews [Software v1.6.2], Zenodo.
  18. Peng, 2021, Reproducible Research: A Retrospective, Annu. Rev. Public Health, 42, 79, 10.1146/annurev-publhealth-012420-105110 · DOI
  19. Barba, L.A. (2018). Terminologies for reproducible research. arXiv.
  20. Sarhan, 2021, SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing, Front. Res. Metr. Anal., 6, 685591, 10.3389/frma.2021.685591 · DOI · Paper page
  21. Haddaway, 2022, A suggested data structure for transparent and repeatable reporting of bibliographic searching, Campbell Syst. Rev., 18, e1288, 10.1002/cl2.1288 · DOI
  22. Butler, 2022, Legal reform to enhance global text and data mining research, Science, 378, 951, 10.1126/science.add6124 · DOI
  23. Jefferson, O.A., Koellhofer, D., Warren, B., and Jefferson, R. (2024, April 25). The Lens MetaRecord and LensID: An Open Identifier System for Aggregated Metadata and Versioning of Knowledge Artefacts, Available online: https://doi.org/10.31229/osf.io/t56yh. · DOI
  24. Priem, J., Piwowar, H., and Orr, R. (2022). Open Alex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv.
  25. Neeleman, 2024, Addressing the challenges of reconstructing systematic reviews datasets: A case study and a noisy label filter procedure, Syst. Rev., 13, 69, 10.1186/s13643-024-02472-w · DOI · Paper page
  26. Warrens, 2019, Kappa coefficients for missing data, Educ. Psychol. Meas., 79, 558, 10.1177/0013164418823249 · DOI
  27. 2022, and J.M. Girard, Weighting schemes and incomplete data: A generalized Bayesian framework for chance-corrected interrater agreement, Psychol. Methods, 27, 1069

Cited by

Metadata

Authors
Lombaers, P., de Bruin, J., & van de Schoot, R.
Venue
Applied Sciences
Publisher
MDPI AG
Published
2024-04-30
Volume
14
Issue
9
Pages
3842
ISSN
2076-3417
DOI
10.3390/app14093842
Cited by
20
References
18
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor)
Funders
  • European Commission (H2020-INNOSUP-2020-02, 406.22.GO.048)
  • Dutch Research Council (H2020-INNOSUP-2020-02, 406.22.GO.048)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.