Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index; 3 of the 22 sources its claims cite were checked for support.

Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders

Open access
Teijema, Jelle Jasper, Hofstee, Laura, Brouwer, Marlies, de Bruin, Jonathan, Ferdinands, Gerbrich, de Boer, Jan, Vizan, Pablo, van den Brand, Sofie, Bockting, Claudi, van de Schoot, Rens, Bagheri, Ayoub · 2023 · Frontiers in Research Metrics and Analytics

Abstract

Introduction: This study examines the performance of active learning-aided systematic reviews using a deep learning-based model compared to traditional machine learning approaches, and explores the potential benefits of model-switching strategies. Methods: Comprising four parts, the study: 1) analyzes the performance and stability of active learning-aided systematic review; 2) implements a convolutional neural network classifier; 3) compares classifier and feature extractor performance; and 4) investigates the impact of model-switching strategies on review performance. Results: Lighter models perform well in early simulation stages, while other models show increased performance in later stages. Model-switching strategies generally improve performance compared to using the default classification model alone. Discussion: The study's findings support the use of model-switching strategies in active learning-based systematic review workflows. It is advised to begin the review with a light model, such as Naïve Bayes or logistic regression, and switch to a heavier classification model based on a heuristic rule when needed.

Overview

CoChat Summary

TL;DR

This paper evaluates active learning-aided systematic reviewing on a large depression-relapse dataset and tests whether switching classification models during screening improves performance. It finds that active learning saves substantial screening time, that lighter models perform well early while heavier neural models help later, and that model-switching generally improves over the default model alone.

Key findings

  • On the Brouwer et al. dataset of 46,376 records with 63 relevant papers (<0.14% relevance), active learning saved on average 92% of screening time (SD = 0.18; min/max = 91.65/92.25), and after screening only 5% of records, 95% of relevant records were found (SD = 0.35; min/max = 95.16/96.77).
  • Excluding the 5% or 10% last-to-find records did not change meta-analysis conclusions overall; one odds-ratio subgroup changed from “just”-significant to “just” not-significant due to one missing study.
  • In Study 3, the best WSS@95% was Logistic Regression + SBERT at 94.21%, finding 95% of relevant records after 587 screened records (1.3% of all records); LR + TF-IDF was close at 94.14% with much lower feature-extraction time (0:00:23.35 vs. 6:27:23.23).
  • In Study 4, switching from Naïve Bayes + TF-IDF to another model almost always increased performance, especially when switching to a different feature extractor; first running a shallow model and then switching to the 17-layer CNN outperformed screening with the CNN alone.

Methods

  • Four simulation studies used the pre-labeled Brouwer et al. depression-relapse dataset (46,376 records, 63 included) and ASReview v0.18 simulation mode.
  • Study 1 used default Naïve Bayes + TF-IDF with 63 runs, one relevant and ten irrelevant prior records, and measured WSS@95%, average time to discovery, and re-ran meta-analyses excluding the 5% and 10% last-to-find records.
  • Studies 2–4 developed a 17-layer CNN with wide Doc2Vec, compared classifiers (NB, LR, RF, SVM, 2-layer NN) and feature extractors (TF-IDF, Doc2Vec, SBERT) on WSS and computational time, and tested model-switching after a heuristic stopping rule (e.g., 50 consecutive irrelevant records).

Limitations

  • The paper states that model selection and performance results are specific to the Brouwer dataset and may not generalize; it recommends a benchmark platform with multiple datasets.
  • The paper states unresolved challenges in re-training frequency, training-time trade-offs, optimal model-switching point, and how content differences affect record findability.
  • The text also makes evident that simulations were terminated after all relevant records were found, which is not representative of real active learning reviews where it is unknown when all relevant records have been found.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/frma.2023.1178181

Cites

36 of 37 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Simulation results of study 1. The absolute amount of relevant publications found is displayed on the y-axis, and the absolute amount of screened publications is on the x-axis. The solid blue lines are a combination of all the Recall curves, representing the relevant records found as a function of the screened publications for each of the 63 ran simulations.
Figure 1 Simulation results of study 1. The absolute amount of relevant publications found is displayed on the y-axis, and the absolute amount of screened publications is on the x-axis. The solid blue lines are a combination of all the Recall curves, representing the relevant records found as a function of the screened publications for each of the 63 ran simulations.CC BY · PubMed Central
Figure 2 Proposed convolutional neural network model. Each element represents a different layer of the neural network. Numbers behind the layer title are settings for that layer.
Figure 2 Proposed convolutional neural network model. Each element represents a different layer of the neural network. Numbers behind the layer title are settings for that layer.CC BY · PubMed Central
Figure 3 Recall curves for each of the simulation runs performed in this study. The x-axis shows the number of screened records. It is cut off after 4,000 records, less than 10% of the total amount of available records. The y-axis represents the found relevant records. The dataset contains 63 relevant records in total, and ten were given as prior knowledge, making the relevant record axis in this figure go up to 53 records. The dotted gray line represents random reading. The colored dotted lines represent the WSS@95% for each simulation. LR, logistic regression; SVM, support vector machine; NB, naïve Bayes; RF, random forest; nn-2-layer, 2 layers deep neural network; CNN, 17 layers convolutional neural network; TF-IDF, term frequency-inverse document frequency; SBERT, Sentence-BERT. Each simulation was cut off after all relevant records were found (n = 4000).
Figure 3 Recall curves for each of the simulation runs performed in this study. The x-axis shows the number of screened records. It is cut off after 4,000 records, less than 10% of the total amount of available records. The y-axis represents the found relevant records. The dataset contains 63 relevant records in total, and ten were given as prior knowledge, making the relevant record axis in this figure go up to 53 records. The dotted gray line represents random reading. The colored dotted lines represent the WSS@95% for each simulation. LR, logistic regression; SVM, support vector machine; NB, naïve Bayes; RF, random forest; nn-2-layer, 2 layers deep neural network; CNN, 17 layers convolutional neural network; TF-IDF, term frequency-inverse document frequency; SBERT, Sentence-BERT. Each simulation was cut off after all relevant records were found (n = 4000).CC BY · PubMed Central
Figure 4 Neural network comparison at WSS@90%. The convolutional neural network with the wider doc2vec implementation and the two-layer neural network with both SBERT and doc2vec as feature extractors. When compared in finding the last record, only the convolutional neural network finds these records before the cutoff.
Figure 4 Neural network comparison at WSS@90%. The convolutional neural network with the wider doc2vec implementation and the two-layer neural network with both SBERT and doc2vec as feature extractors. When compared in finding the last record, only the convolutional neural network finds these records before the cutoff.CC BY · PubMed Central
Figure 5 Rank order cohesion correlation matrix. This figure shows how similar or different the order can be between models.
Figure 5 Rank order cohesion correlation matrix. This figure shows how similar or different the order can be between models.CC BY · PubMed Central
Figure 6 Study 4 simulation results. (A, C) Show the recall curve for the simulation starting as Naïve Bayes (A) and logistic regression (C) before and after switching. (B, D) Show the recall plots of the other models after switching.
Figure 6 Study 4 simulation results. (A, C) Show the recall curve for the simulation starting as Naïve Bayes (A) and logistic regression (C) before and after switching. (B, D) Show the recall plots of the other models after switching.CC BY · PubMed Central
Figure 7 Study 4 switching results for the 17-layer CNN model combined with (A) Naïve Bayes and (B) logistic regression. The value on the y-axis is the total number of records found. The RRF@X% represents the number of found relevant records after screening X%.
Figure 7 Study 4 switching results for the 17-layer CNN model combined with (A) Naïve Bayes and (B) logistic regression. The value on the y-axis is the total number of records found. The RRF@X% represents the number of found relevant records after screening X%.CC BY · PubMed Central
Figure 8 (A, B) The relative performance of classifiers after switching. (A) Shows the performance of switching from Naïve Bayes to other models (x). (B) Shows logistic regression to other models (x).
Figure 8 (A, B) The relative performance of classifiers after switching. (A) Shows the performance of switching from Naïve Bayes to other models (x). (B) Shows logistic regression to other models (x).CC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 2 Not valid · 18 Not checked · 87 Own results and interpretations

63 more claims

Create a free account to see all 109 claims and to verify the 19 references still open.

See all claims — free account
  • Probably not a substantial claim. 18
  • It may state two separate claims as one. 28

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 35 references

Research works printed but not linked: 9

  1. Ferdinands, 2021, Scripts and Output for the Simulation Study Determining the Time to Discovery for the Depression Data (1, 0.).
  2. Goodfellow, 2016, Deep learning: MIT press;
  3. Parker, 2011, English Gigaword
  4. Teijema, ASReview CNN 17 layer model plugin. v1, 0.2 ed.
  5. Teijema, ASReview wide doc2vec plugin. v0, 1.2 ed.
  6. Teijema, ASReview model switcher plugin. v1, 0.2 ed.
  7. van de Brand, 2021, A Systematic Review on Studies Evaluating the Performance of Active Learning Compared to Human Reading for Systematic Review Data
  8. van de Schoot, ASReview: Active learning for systematic reviews [Software]
  9. van den Brand, 2021, Scripts for Post-Processing Mega-Meta Screening Results. v1, 0.1 ed.

Cited by

See all 36

Metadata

Authors
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., & Bagheri, A.
Affiliations
  • Utrecht University (NL)
  • Amsterdam University Medical Centers (NL)
Venue
Frontiers in Research Metrics and Analytics
Published
2023-05-16
Volume
8
Pages
1178181-1178181
ISSN
2504-0537 (ISSN-L)
OpenAlex
W4376645833
Cited by
36
References
35
Citations by year
2026: 9 · 2025: 10 · 2024: 13 · 2023: 4
FWCI
2.55
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Research Metrics and Analytics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Universiteit van Amsterdam
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
active learning (machine learning), systematic reviews, model switching, convolutional neural networks, feature extraction, depression onset, depression maintenance, depression relapse

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders. https://app.cochat.ai/papers/10.3389/frma.2023.1178181
The paper
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., & Bagheri, A. (2023). Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders. Frontiers in Research Metrics and Analytics. https://doi.org/10.3389/frma.2023.1178181

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.