Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.

Active learning-based Systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders

Open access
Teijema, Jelle Jasper, Hofstee, Laura, Brouwer, Marlies, de Bruin, Jonathan, Ferdinands, Gerbrich, de Boer, Jan, Siso, Pablo Vizan, van den Brand, Sofie Arsenia Gabriëlla Eleonora, Bockting, Claudi, van de Schoot, Rens, Bagheri, Ayoub · 2022

Abstract

Systematic reviews and meta-analyses are top of the bill in research. However, the screening phase requires an enormous effort in reading and labeling thousands of papers identified via systematic search. Active learning-aided systematic reviewing offers a solution by combining machine learning algorithms with user input to reduce screening load. This study explores the performance of these algorithms and different ways to apply them. This study is divided into four studies evaluating and improving this active learning pipeline. First, the performance and stability of the active learning pipeline were assessed via simulations and re-analysis of the outcome. Secondly, a convolutional neural network was developed to improve upon available machine learning algorithms. Thirdly, the performance of different algorithm combinations was tested and compared. Finally, algorithm-switching models were built for increased performance. The study concludes with proposals for improving active learning-aided systematic reviews based on combinations of the four studies. It was found that switching models can outperform the currently used models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.31234/osf.io/t7bpd

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1A Visualization of the relevant records from the Brouwer et al. dataset. Using (26).
Figure 1A Visualization of the relevant records from the Brouwer et al. dataset. Using (26).CC-BY · From the PDF
Figure 1B Visualization of the irrelevant records from the Brouwer et al. dataset. Using (26).
Figure 1B Visualization of the irrelevant records from the Brouwer et al. dataset. Using (26).CC-BY · From the PDF
Figure 2 Simulation results of study 1. The percentage of relevant publications found is displayed on the y-axis, and the percentage of screened publications is on the x-axis. The solid blue lines are the Recall curves, representing the relevant records found as a function of the screened publications for each of the 63 runs.
Figure 2 Simulation results of study 1. The percentage of relevant publications found is displayed on the y-axis, and the percentage of screened publications is on the x-axis. The solid blue lines are the Recall curves, representing the relevant records found as a function of the screened publications for each of the 63 runs.CC-BY · From the PDF
Figure 3 Proposed convolutional neural network model. Each element represents a different layer of the neural network. Numbers behind the layer title are settings for that layer.
Figure 3 Proposed convolutional neural network model. Each element represents a different layer of the neural network. Numbers behind the layer title are settings for that layer.CC-BY · From the PDF
Figure 4 recall curves for each of the simulation runs performed in this study. The x-axis shows the number of screened records. It is cut off after 4000 records, less than 10% of the total amount of available records. The y-axis represents the found relevant records. The dataset contains 63 relevant records in total, and ten were given as prior knowledge, making the relevant record axis in this figure go up to 53 records. The dotted grey line represents random reading. The colored dotted lines represent the WSS@95% for each simulation. LR, logistic regression; SVM, support vector machine; NB, naïve Bayes; RF, random forest; nn-2-layer, 2 layers deep neural network; CNN, 17 layers convolutional neural network; TF-IDF, term frequency-inverse document frequency; SBERT, Sentence-BERT. Each simulation was cut off after all relevant records were found (n=4000).
Figure 4 recall curves for each of the simulation runs performed in this study. The x-axis shows the number of screened records. It is cut off after 4000 records, less than 10% of the total amount of available records. The y-axis represents the found relevant records. The dataset contains 63 relevant records in total, and ten were given as prior knowledge, making the relevant record axis in this figure go up to 53 records. The dotted grey line represents random reading. The colored dotted lines represent the WSS@95% for each simulation. LR, logistic regression; SVM, support vector machine; NB, naïve Bayes; RF, random forest; nn-2-layer, 2 layers deep neural network; CNN, 17 layers convolutional neural network; TF-IDF, term frequency-inverse document frequency; SBERT, Sentence-BERT. Each simulation was cut off after all relevant records were found (n=4000).CC-BY · From the PDF
Figure 5 Neural network comparison at WSS@90%. The convolutional neural network with the wider doc2vec implementation and the two-layer neural network with both SBERT and doc2vec as feature extractors. When compared in finding the last record, only the convolutional neural network finds these records before the cutoff.
Figure 5 Neural network comparison at WSS@90%. The convolutional neural network with the wider doc2vec implementation and the two-layer neural network with both SBERT and doc2vec as feature extractors. When compared in finding the last record, only the convolutional neural network finds these records before the cutoff.CC-BY · From the PDF
Figure 6 Rank order cohesion correlation matrix. This figure shows how similar or different the order can be between models.
Figure 6 Rank order cohesion correlation matrix. This figure shows how similar or different the order can be between models.CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Figure 7 Study 4 simulation results. Panels A and C show the recall curve for the simulation starting as Naïve Bayes (A) and logistic regression (C) before and after switching. Panels (B) and (D) show the recall plots of the other models after switching.
Figure 7 Study 4 simulation results. Panels A and C show the recall curve for the simulation starting as Naïve Bayes (A) and logistic regression (C) before and after switching. Panels (B) and (D) show the recall plots of the other models after switching.CC-BY · From the PDF
Figure 8 Study 4 switching results for the 17-layer CNN model combined with (A) Naïve Bayes and (B) logistic regression. The value on the y-axis is the total number of records found. The RRF@X% represents the number of found relevant records after screening X%
Figure 8 Study 4 switching results for the 17-layer CNN model combined with (A) Naïve Bayes and (B) logistic regression. The value on the y-axis is the total number of records found. The RRF@X% represents the number of found relevant records after screening X%CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Image 12
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 24 references

Research works printed but not linked: 7

  1. [2]2. Settles B. Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. 2012;6(1):1-114.
  2. [3]3. van de Brand S, van de Schoot R. A Systematic Review on Studies Evaluating the Performance of Active Learning Compared to Human Reading for Systematic Review Data. OSF 2021.
  3. [7]7. Goodfellow I, Bengio Y, Courville A. Deep learning: MIT press; 2016.
  4. [9]9. Collobert R, Weston J, editors. A unified architecture for natural language processing: Deep neural networks with multitask learning. Proceedings of the 25th international conference on Machine learning; 2008.
  5. [13]13. Parker R, Graff D, Kong J, Chen K, Maeda K. English gigaword fifth edition, linguistic data consortium. Google Scholar. 2011.
  6. [24]24. Brouwer M, Ferdinands G, van den Brand S, de Boer J, de Bruin J, Schulte-Frankenfeld P, et al. Systematic review data from "Psychological theories of depressive relapse and recurrence" (Brouwer et al., 2019). 2021.
  7. [26]26. Ma Y, Van den Brand S, Van de Schoot R, De Bruin J. Wordclouds: A tool to create a visual impression of the verbal content within a systematic review dataset. v0.4 ed: Zenodo; 2021.

Cited by

See all 15

Metadata

Authors
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Siso, P. V., van den Brand, S. A. G. E., Bockting, C., van de Schoot, R., & Bagheri, A.
Affiliations
  • Utrecht University (NL)
  • Amsterdam University Medical Centers (NL)
Published
2022-07-15
OpenAlex
W4285593393
Cited by
14
References
24
Citations by year
2025: 2 · 2024: 5 · 2023: 7
Type
posted content
Language
en
Open access
Locations
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
Funders
Universiteit van Amsterdam
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
Keywords
active learning (machine learning), systematic review screening, algorithm switching, convolutional neural networks, depression onset, depression maintenance, depression relapse, meta-analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Active learning-based Systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders. https://app.cochat.ai/papers/10.31234/osf.io/t7bpd
The paper
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Siso, P. V., van den Brand, S. A. G. E., Bockting, C., van de Schoot, R., & Bagheri, A. (2022). Active learning-based Systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders. https://doi.org/10.31234/osf.io/t7bpd

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.