Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 37 of the 49 sources its claims cite were checked for support.

Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research

Open access

Abstract

Abstract Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the sensitivity, specificity, and estimated time savings of several learning algorithms and heuristic stopping criteria. The results showed, on average, a 58% ( SD = 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days ( SD = 1.80). The learning algorithm random forests with sentence bidirectional encoder representations from transformers outperformed other algorithms. This finding emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after classifying 20% of records and consecutively classifying 5% of irrelevant papers yielded the most significant gains in terms of specificity ( M = 42%, SD = 28%). However, the performance of the heuristic stopping criteria depended on the learning algorithm used and the length and proportion of relevant papers in an abstract collection. Our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.

Overview

CoChat Summary

TL;DR

This retrospective simulation of 27 education and educational psychology systematic reviews evaluated machine-learning screening algorithms and heuristic stopping criteria in ASReview. It found active learning reduced irrelevant-record screening and that heuristic rules could retrieve 95% of relevant abstracts, but performance depended on algorithm and dataset characteristics.

Key findings

  • Across algorithms, abstract screening with learning algorithms reduced screening workload of irrelevant records by 58% (SD = 19%) on average and saved an estimated 1.66 days (SD = 1.80); the abstract reports RF+SBERT outperformed other algorithms, while the results table shows LR+SBERT had the highest mean specificity@95% (65%, SD = 18%) and ETS@95% (1.80 days, SD = 1.94).
  • For heuristic stopping, 95% of relevant abstracts could be retrieved: time-based screening required reviewing an average of 70% of abstracts (saving 0.86 days, SD = 0.73; avoiding 36%, SD = 6%, of irrelevant abstracts); data-driven stopping required classifying an average of 7% consecutive irrelevant abstracts (saving 1.04 days, SD = 1.47; avoiding 38%, SD = 27%, of irrelevant abstracts).
  • The mixed rule stopping after 20% of records screened and 5% consecutive irrelevant records (group E) yielded the highest specificity among mixed rules (M = 42%, SD = 28%) while maintaining ≥95% sensitivity; RF+SBERT had highest average sensitivity (92%, SD = 8%) and specificity (52%, SD = 19%) across mixed strategies, and SVM+Doc2Vec lowest (87%, SD = 13%; 44%, SD = 20%).
  • Exploratory correlations: database length positively correlated with model specificity (ρ = .35, 95% CI [0.22, 0.48]); proportion of relevant studies negatively correlated with specificity (ρ = −.65, 95% CI [−0.72, −0.58]).

Methods

  • Retrospective screening simulation using 27 abstract collections from systematic reviews/meta-analyses in education and educational psychology; datasets averaged 2,738 studies (SD = 2,382), with 18.75% (SD = 15.89%) selected for full-text screening.
  • Used ASReview simulation mode with 10 learning algorithms combining feature extraction (Doc2Vec, SBERT, TF-IDF) and classifiers (NB, LR, RF, SVM), default dynamic resampling and certainty-based sampling, yielding 270 simulation runs; initial training set was one randomly selected included and one excluded study, held constant.
  • Evaluated sensitivity, specificity, and estimated time savings at 95% sensitivity for algorithms, and for time-based (10–100% screening intervals), data-driven (1–10% consecutive irrelevant), and mixed heuristic stopping criteria; analyses used R 4.2.3 and ASReview Makita.

Limitations

  • Stated: the sample was heterogeneous but neither exhaustive nor representative; many included reviews were unpublished/not peer reviewed, so caution is needed in extrapolating to all education/educational psychology systematic reviews.
  • Stated: the initial training set was randomly selected and held constant across simulation conditions; future work should vary/expand it, and the study should be extended to other algorithms and statistical stopping criteria.
  • Stated/evident: heuristic stopping rules were developed with knowledge of true relevant records and their performance depended on database size, proportion of relevant papers, topic complexity, and algorithm, so they should not be generalized to databases where the number of relevant records is unknown.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10648-024-09862-5

Cites

91 of 91 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

26 Valid · 3 Not valid · 6 Not checked · 88 Own results and interpretations

74 more claims · 10 with their cited sources checked

Create a free account to see all 123 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 27
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Research works printed but not linked: 3

  1. Fütterer, T., Tschönhens, F., Scherer, R., Dickhäuser, O., & Ruiz-Primo, M. A. (2023). Teachers’ self-assessment ability—A systematic literature review and meta-analysis. [Manuscript in preparation]. Faculty of Economic and Social Sciences. University of Tübingen.
  2. Veletić, J., Rodriguez-Mejia, A. M., & Olsen, R. V. (2023). A systematic literature review of the Teaching and Learning International Survey (TALIS) research. Faculty of Educational Sciences. University of Oslo
  3. World Health Organization. (2019). The ICD-10 classification of mental and behavioural disorders: Diagnostic criteria for research. World Health Organization.

Cited by

See all 75

Metadata

Authors
Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R.
Affiliations
  • University of Oslo (NO)
  • University of Tübingen (DE)
  • University of Copenhagen (DK)
  • Helmut Schmidt University (DE)
Corresponding author
Campos, Diego G.
Venue
Educational Psychology Review
Published
2024-02-08
Volume
36
Issue
1
ISSN
  • 1040-726X (ISSN-L)
  • 1573-336X
OpenAlex
W4391653771
Cited by
75
References
74
Citations by year
2026: 33 · 2025: 32 · 2024: 10
FWCI
9.43
Type
journal article
Language
en
Open access
Licence
Locations
  • Educational Psychology Review (publishedVersion, cc-by)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
  • NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
Funders
University of Oslo
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
systematic review screening, abstract screening, random forest, BERT, educational research, educational psychology, sensitivity and specificity, time savings

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.