The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 37 of the 49 sources its claims cite were checked for support.
Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research
Open accessAbstract
Abstract Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the sensitivity, specificity, and estimated time savings of several learning algorithms and heuristic stopping criteria. The results showed, on average, a 58% ( SD = 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days ( SD = 1.80). The learning algorithm random forests with sentence bidirectional encoder representations from transformers outperformed other algorithms. This finding emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after classifying 20% of records and consecutively classifying 5% of irrelevant papers yielded the most significant gains in terms of specificity ( M = 42%, SD = 28%). However, the performance of the heuristic stopping criteria depended on the learning algorithm used and the length and proportion of relevant papers in an abstract collection. Our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.
Overview
CoChat Summary
TL;DR
This retrospective simulation of 27 education and educational psychology systematic reviews evaluated machine-learning screening algorithms and heuristic stopping criteria in ASReview. It found active learning reduced irrelevant-record screening and that heuristic rules could retrieve 95% of relevant abstracts, but performance depended on algorithm and dataset characteristics.
Key findings
- Across algorithms, abstract screening with learning algorithms reduced screening workload of irrelevant records by 58% (SD = 19%) on average and saved an estimated 1.66 days (SD = 1.80); the abstract reports RF+SBERT outperformed other algorithms, while the results table shows LR+SBERT had the highest mean specificity@95% (65%, SD = 18%) and ETS@95% (1.80 days, SD = 1.94).
- For heuristic stopping, 95% of relevant abstracts could be retrieved: time-based screening required reviewing an average of 70% of abstracts (saving 0.86 days, SD = 0.73; avoiding 36%, SD = 6%, of irrelevant abstracts); data-driven stopping required classifying an average of 7% consecutive irrelevant abstracts (saving 1.04 days, SD = 1.47; avoiding 38%, SD = 27%, of irrelevant abstracts).
- The mixed rule stopping after 20% of records screened and 5% consecutive irrelevant records (group E) yielded the highest specificity among mixed rules (M = 42%, SD = 28%) while maintaining ≥95% sensitivity; RF+SBERT had highest average sensitivity (92%, SD = 8%) and specificity (52%, SD = 19%) across mixed strategies, and SVM+Doc2Vec lowest (87%, SD = 13%; 44%, SD = 20%).
- Exploratory correlations: database length positively correlated with model specificity (ρ = .35, 95% CI [0.22, 0.48]); proportion of relevant studies negatively correlated with specificity (ρ = −.65, 95% CI [−0.72, −0.58]).
Methods
- Retrospective screening simulation using 27 abstract collections from systematic reviews/meta-analyses in education and educational psychology; datasets averaged 2,738 studies (SD = 2,382), with 18.75% (SD = 15.89%) selected for full-text screening.
- Used ASReview simulation mode with 10 learning algorithms combining feature extraction (Doc2Vec, SBERT, TF-IDF) and classifiers (NB, LR, RF, SVM), default dynamic resampling and certainty-based sampling, yielding 270 simulation runs; initial training set was one randomly selected included and one excluded study, held constant.
- Evaluated sensitivity, specificity, and estimated time savings at 95% sensitivity for algorithms, and for time-based (10–100% screening intervals), data-driven (1–10% consecutive irrelevant), and mixed heuristic stopping criteria; analyses used R 4.2.3 and ASReview Makita.
Limitations
- Stated: the sample was heterogeneous but neither exhaustive nor representative; many included reviews were unpublished/not peer reviewed, so caution is needed in extrapolating to all education/educational psychology systematic reviews.
- Stated: the initial training set was randomly selected and held constant across simulation conditions; future work should vary/expand it, and the study should be extended to other algorithms and statistical stopping criteria.
- Stated/evident: heuristic stopping rules were developed with knowledge of true relevant records and their performance depended on database size, proportion of relevant papers, topic complexity, and algorithm, so they should not be generalized to databases where the number of relevant records is unknown.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
26 Valid · 3 Not valid · 6 Not checked · 88 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“The former estimates the stopping criteria based on the derivation of an estimate of the total number of relevant papers within an initial training set and is considered more reliable than the latter (Cormack & Grossman, 2016; van Haastrecht et al., 2021).”
What the sources say
- Disputedcites Cormack & Grossman, 2016 sourceExperimental — accuracy not yet measured
The Introduction describes the statistical target method as provably reliable, but says the geometric knee method has similar empirical reliability and the budget method has superior empirical reliability. The Discussion likewise reports that the budget method was more reliable than the target method on the test collections. The paper therefore does not support a general ranking of statistical stopping criteria above heuristic ones. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Supportedcites van Haastrecht et al., 2021 source
Section 3.3 describes stopping criteria based on estimating the total number of relevant papers in the starting set. It says these criteria solve issues with simpler criteria, which it does not consider best practice, and recommends investigating more mathematically grounded approaches. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“as their development depends on knowledge of the true relevant records (Callaghan & Müller-Hansen, 2020; Hamel et al., 2021).”
What the sources say
- Partly supportedcites Callaghan & Müller-Hansen, 2020 sourceExperimental — accuracy not yet measured
In the Methods review (page not available), the paper says that developing one pragmatic stopping criterion—stopping after half the documents have been screened—depended on knowing the true relevance values. It describes heuristic criteria separately as stopping after a specified run of irrelevant records, so it does not support the claim that heuristic stopping rules generally depend on that knowledge. (LLM, conf=0.87)
- Disputedcites Hamel et al., 2021 sourceExperimental — accuracy not yet measured
Step 7, “Truncation of screening” (page not provided), describes a heuristic stopping rule based on a run of consecutively reviewed irrelevant records, not knowledge of all truly relevant records. It discusses unknown true recall separately in connection with a more complex, estimated-recall stopping decision. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“In contrast, the conceptual clarity of constructs commonly used in education and educational psychology has been recently criticized (Bringmann et al., 2022; Flake & Fried, 2020; Marsh et al., 2019).”
What the sources say
- Not checked — no full text of the sourcecites Bringmann et al., 2022 source
- Not checked — no full text of the sourcecites Flake & Fried, 2020 source
- Supportedcites Marsh et al., 2019 source
In “Generalizability of Conclusions: Limitations and Directions for Further Research” (page not provided), the paper describes a lack of clarity in key constructs underpinning educational psychology research. This directly supports the cited claim. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In contrast, Callaghan and Müller-Hansen (2020) found that data-driven approaches could not find 95% of all relevant records in 39% of the 20 systematic reviews analyzed.”
What the sources say
- Partly supportedcites Callaghan and Müller-Hansen (2020) sourceExperimental — accuracy not yet measured
The Results report that the baseline sampling stopping criterion missed the 95% recall target in 39.67% of cases. The Methods (Evaluation) say each criterion was tested over 100 iterations per dataset, so this is not a finding that 39% of 20 reviews failed. The Results also report that the paper’s random and ranked sampling criteria achieved the target in more than 95% of cases; the claim overgeneralizes the baseline result to data-driven approaches as a whole. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, existing tools lack the flexibility to handle the wide range of classifiers and feature extraction techniques that can be implemented in ML screening tools effectively(van de Schoot et al., 2021).”
What the sources say
- Partly supportedcites van de Schoot et al., 2021 sourceExperimental — accuracy not yet measured
In the passage labelled “Not available” (page unavailable), the paper says that, to its knowledge, existing tools lack flexibility for the range of concepts encountered in screening and that successful systems need a wide range of classifier types. The “Software implementation for ASReview” section lists multiple feature-extraction techniques in ASReview, but does not establish that existing tools lack flexibility specifically for those techniques. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
74 more claims · 10 with their cited sources checked
Create a free account to see all 123 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 27
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Mixed Strategy
“a positive correlation between the sensitivity achieved by the mixed-based heuristic stopping criteria and the number of relevant studies in an abstract collection in groups C, F, I, and G”
§ Mixed Strategy
“ρ = - 0.50, 95% CI [- 0.77, - 0.09]”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Time Strategy
“| 100% | 100% | 100% | 0% | 0% | 0% | 1% | 0 | 0 | 0 |”
§ Data Analysis
“Specificity (true negative rate): The proportion of studies in an abstract collection that would not require screening by reviewers when using learning algorithms out of the total deemed irrelevant by the authors of the systematic review”
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2×
“In contrast, word embed-dings capture the semantic meaning of words by representing them as dense vectors in a continuous space, trained on large text corpora (Mikolov, Chen, et al., 2013; Mikolov, Sutskever, et al., 2013).”
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013) · Wiley Series in Probability and Statistics · doi:10.1002/9781118548387 · Cited by 13,861
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 12×
“Conversely, the RF+SBERT model had a higher frequency of producing the highest specificity values.”
InfluentialClassification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 1×
“BoW and TF-IDF are conventional approaches representing text data as a vector of word or term frequencies, respectively (Harris, 1954; Salton & Buckley, 1988).”
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,385
Machine learning: Algorithms, real-world applications and research directions
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00592-x · Cited by 5,583
Research works printed but not linked: 3
- Fütterer, T., Tschönhens, F., Scherer, R., Dickhäuser, O., & Ruiz-Primo, M. A. (2023). Teachers’ self-assessment ability—A systematic literature review and meta-analysis. [Manuscript in preparation]. Faculty of Economic and Social Sciences. University of Tübingen.
- Veletić, J., Rodriguez-Mejia, A. M., & Olsen, R. V. (2023). A systematic literature review of the Teaching and Learning International Survey (TALIS) research. Faculty of Educational Sciences. University of Oslo
- World Health Organization. (2019). The ICD-10 classification of mental and behavioural disorders: Diagnostic criteria for research. World Health Organization.
Cited by
Enhancing peer assessment with artificial intelligence
Topping, K. J., Gehringer, E. F., Khosravi, H., Gudipati, S., Jadhav, K., & Susarla, S. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00501-1 · Cited by 104
Bourke, M., Wang, H., McNaughton, S. A., Thomas, G., Firth, J., Trott, M., & Cairney, J. (2025) · Clinical Psychology Review · doi:10.1016/j.cpr.2025.102585 · Cited by 75
The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence
Slattery, P., Saeri, A. K., Grundy, E. A. C., Graham, J., Noetel, M., Uuk, R., Dao, J., Pour, S., Casper, S., & Thompson, N. (2026) · Patterns · doi:10.1016/j.patter.2026.101517 · Cited by 255
“Based on a review of relevant literature (van de Schoot et al 42 ; e.g., Campos et al. 44 ), we use the Oracle model with the following setup: a fully connected neural network (two hidden layers) model as the classifier and sBert as the feature extractor, maximum as the query strategy, and dynamic resampling (double)…”
InfluentialKönig, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1762 · Cited by 39
“16,19,26,38,39 Consequently, performance variations may arise when using AI-AS for abstract collections from other research domains.”
InfluentialSelf-citationASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts
de Bruin, J., Lombaers, P., Kaandorp, C., Teijema, J., van der Kuil, T., Yazan, B., Dong, A., & van de Schoot, R. (2025) · Patterns · doi:10.1016/j.patter.2025.101318 · Cited by 27
“32,39,43 There is also the global stopping condition next to the cycle’s stopping condition, for example, when the end of data is reached or if no more agents are available.”
Metadata
- Authors
- Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R.
- Affiliations
- University of Oslo (NO)
- University of Tübingen (DE)
- University of Copenhagen (DK)
- Helmut Schmidt University (DE)
- Corresponding author
- Campos, Diego G.
- ORCID
- Venue
- Educational Psychology Review
- Published
- 2024-02-08
- Volume
- 36
- Issue
- 1
- ISSN
- 1040-726X (ISSN-L)
- 1573-336X
- OpenAlex
- W4391653771
- Cited by
- 75
- References
- 74
- Citations by year
- 2026: 33 · 2025: 32 · 2024: 10
- FWCI
- 9.43
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Educational Psychology Review (publishedVersion, cc-by)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Funders
- University of Oslo
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- systematic review screening, abstract screening, random forest, BERT, educational research, educational psychology, sensitivity and specificity, time savings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research. https://app.cochat.ai/papers/10.1007/s10648-024-09862-5
- The paper
- Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2024). Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research. Educational Psychology Review. https://doi.org/10.1007/s10648-024-09862-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.