The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; 37 of the 49 sources its claims cite were checked for support.
Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research
Open accessAbstract
Abstract Systematic reviews and meta-analyses are crucial for advancing research, yet they are time-consuming and resource-demanding. Although machine learning and natural language processing algorithms may reduce this time and these resources, their performance has not been tested in education and educational psychology, and there is a lack of clear information on when researchers should stop the reviewing process. In this study, we conducted a retrospective screening simulation using 27 systematic reviews in education and educational psychology. We evaluated the sensitivity, specificity, and estimated time savings of several learning algorithms and heuristic stopping criteria. The results showed, on average, a 58% ( SD = 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days ( SD = 1.80). The learning algorithm random forests with sentence bidirectional encoder representations from transformers outperformed other algorithms. This finding emphasizes the importance of incorporating semantic and contextual information during feature extraction and modeling in the screening process. Furthermore, we found that 95% of all relevant abstracts within a given dataset can be retrieved using heuristic stopping rules. Specifically, an approach that stops the screening process after classifying 20% of records and consecutively classifying 5% of irrelevant papers yielded the most significant gains in terms of specificity ( M = 42%, SD = 28%). However, the performance of the heuristic stopping criteria depended on the learning algorithm used and the length and proportion of relevant papers in an abstract collection. Our study provides empirical evidence on the performance of machine learning screening algorithms for abstract screening in systematic reviews in education and educational psychology.
Overview
CoChat Summary
TL;DR
This retrospective simulation of 27 education and educational psychology systematic reviews evaluated machine-learning screening algorithms and heuristic stopping criteria in ASReview. It found active learning reduced irrelevant-record screening and that heuristic rules could retrieve 95% of relevant abstracts, but performance depended on algorithm and dataset characteristics.
Key findings
- Across algorithms, abstract screening with learning algorithms reduced screening workload of irrelevant records by 58% (SD = 19%) on average and saved an estimated 1.66 days (SD = 1.80); the abstract reports RF+SBERT outperformed other algorithms, while the results table shows LR+SBERT had the highest mean specificity@95% (65%, SD = 18%) and ETS@95% (1.80 days, SD = 1.94).
- For heuristic stopping, 95% of relevant abstracts could be retrieved: time-based screening required reviewing an average of 70% of abstracts (saving 0.86 days, SD = 0.73; avoiding 36%, SD = 6%, of irrelevant abstracts); data-driven stopping required classifying an average of 7% consecutive irrelevant abstracts (saving 1.04 days, SD = 1.47; avoiding 38%, SD = 27%, of irrelevant abstracts).
- The mixed rule stopping after 20% of records screened and 5% consecutive irrelevant records (group E) yielded the highest specificity among mixed rules (M = 42%, SD = 28%) while maintaining ≥95% sensitivity; RF+SBERT had highest average sensitivity (92%, SD = 8%) and specificity (52%, SD = 19%) across mixed strategies, and SVM+Doc2Vec lowest (87%, SD = 13%; 44%, SD = 20%).
- Exploratory correlations: database length positively correlated with model specificity (ρ = .35, 95% CI [0.22, 0.48]); proportion of relevant studies negatively correlated with specificity (ρ = −.65, 95% CI [−0.72, −0.58]).
Methods
- Retrospective screening simulation using 27 abstract collections from systematic reviews/meta-analyses in education and educational psychology; datasets averaged 2,738 studies (SD = 2,382), with 18.75% (SD = 15.89%) selected for full-text screening.
- Used ASReview simulation mode with 10 learning algorithms combining feature extraction (Doc2Vec, SBERT, TF-IDF) and classifiers (NB, LR, RF, SVM), default dynamic resampling and certainty-based sampling, yielding 270 simulation runs; initial training set was one randomly selected included and one excluded study, held constant.
- Evaluated sensitivity, specificity, and estimated time savings at 95% sensitivity for algorithms, and for time-based (10–100% screening intervals), data-driven (1–10% consecutive irrelevant), and mixed heuristic stopping criteria; analyses used R 4.2.3 and ASReview Makita.
Limitations
- Stated: the sample was heterogeneous but neither exhaustive nor representative; many included reviews were unpublished/not peer reviewed, so caution is needed in extrapolating to all education/educational psychology systematic reviews.
- Stated: the initial training set was randomly selected and held constant across simulation conditions; future work should vary/expand it, and the study should be extended to other algorithms and statistical stopping criteria.
- Stated/evident: heuristic stopping rules were developed with knowledge of true relevant records and their performance depended on database size, proportion of relevant papers, topic complexity, and algorithm, so they should not be generalized to databases where the number of relevant records is unknown.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
26 Valid · 3 Not valid · 6 Not checked · 88 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“The former estimates the stopping criteria based on the derivation of an estimate of the total number of relevant papers within an initial training set and is considered more reliable than the latter (Cormack & Grossman, 2016; van Haastrecht et al., 2021).”
What the sources say
- Disputedcites Cormack & Grossman, 2016 sourceExperimental — accuracy not yet measured
The Introduction describes the statistical target method as provably reliable, but says the geometric knee method has similar empirical reliability and the budget method has superior empirical reliability. The Discussion likewise reports that the budget method was more reliable than the target method on the test collections. The paper therefore does not support a general ranking of statistical stopping criteria above heuristic ones. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Supportedcites van Haastrecht et al., 2021 source
Section 3.3 describes stopping criteria based on estimating the total number of relevant papers in the starting set. It says these criteria solve issues with simpler criteria, which it does not consider best practice, and recommends investigating more mathematically grounded approaches. (LLM, conf=0.90)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“as their development depends on knowledge of the true relevant records (Callaghan & Müller-Hansen, 2020; Hamel et al., 2021).”
What the sources say
- Partly supportedcites Callaghan & Müller-Hansen, 2020 sourceExperimental — accuracy not yet measured
In the Methods review (page not available), the paper says that developing one pragmatic stopping criterion—stopping after half the documents have been screened—depended on knowing the true relevance values. It describes heuristic criteria separately as stopping after a specified run of irrelevant records, so it does not support the claim that heuristic stopping rules generally depend on that knowledge. (LLM, conf=0.87)
- Disputedcites Hamel et al., 2021 sourceExperimental — accuracy not yet measured
Step 7, “Truncation of screening” (page not provided), describes a heuristic stopping rule based on a run of consecutively reviewed irrelevant records, not knowledge of all truly relevant records. It discusses unknown true recall separately in connection with a more complex, estimated-recall stopping decision. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“In contrast, the conceptual clarity of constructs commonly used in education and educational psychology has been recently criticized (Bringmann et al., 2022; Flake & Fried, 2020; Marsh et al., 2019).”
What the sources say
- Not checked — no full text of the sourcecites Bringmann et al., 2022 source
- Not checked — no full text of the sourcecites Flake & Fried, 2020 source
- Supportedcites Marsh et al., 2019 source
In “Generalizability of Conclusions: Limitations and Directions for Further Research” (page not provided), the paper describes a lack of clarity in key constructs underpinning educational psychology research. This directly supports the cited claim. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In contrast, Callaghan and Müller-Hansen (2020) found that data-driven approaches could not find 95% of all relevant records in 39% of the 20 systematic reviews analyzed.”
What the sources say
- Partly supportedcites Callaghan and Müller-Hansen (2020) sourceExperimental — accuracy not yet measured
The Results report that the baseline sampling stopping criterion missed the 95% recall target in 39.67% of cases. The Methods (Evaluation) say each criterion was tested over 100 iterations per dataset, so this is not a finding that 39% of 20 reviews failed. The Results also report that the paper’s random and ranked sampling criteria achieved the target in more than 95% of cases; the claim overgeneralizes the baseline result to data-driven approaches as a whole. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, existing tools lack the flexibility to handle the wide range of classifiers and feature extraction techniques that can be implemented in ML screening tools effectively(van de Schoot et al., 2021).”
What the sources say
- Partly supportedcites van de Schoot et al., 2021 sourceExperimental — accuracy not yet measured
In the passage labelled “Not available” (page unavailable), the paper says that, to its knowledge, existing tools lack flexibility for the range of concepts encountered in screening and that successful systems need a wide range of classifier types. The “Software implementation for ASReview” section lists multiple feature-extraction techniques in ASReview, but does not establish that existing tools lack flexibility specifically for those techniques. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
74 more claims · 10 with their cited sources checked
Create a free account to see all 123 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 27
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Mixed Strategy
“a positive correlation between the sensitivity achieved by the mixed-based heuristic stopping criteria and the number of relevant studies in an abstract collection in groups C, F, I, and G”
§ Mixed Strategy
“ρ = - 0.50, 95% CI [- 0.77, - 0.09]”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Time Strategy
“| 100% | 100% | 100% | 0% | 0% | 0% | 1% | 0 | 0 | 0 |”
§ Data Analysis
“Specificity (true negative rate): The proportion of studies in an abstract collection that would not require screening by reviewers when using learning algorithms out of the total deemed irrelevant by the authors of the systematic review”
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2×
“In contrast, word embed-dings capture the semantic meaning of words by representing them as dense vectors in a continuous space, trained on large text corpora (Mikolov, Chen, et al., 2013; Mikolov, Sutskever, et al., 2013).”
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013) · Wiley Series in Probability and Statistics · doi:10.1002/9781118548387 · Cited by 13,861
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 12×
“Conversely, the RF+SBERT model had a higher frequency of producing the highest specificity values.”
InfluentialClassification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 1×
“BoW and TF-IDF are conventional approaches representing text data as a vector of word or term frequencies, respectively (Harris, 1954; Salton & Buckley, 1988).”
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,385
Machine learning: Algorithms, real-world applications and research directions
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00592-x · Cited by 5,583
Harris, Z. (1954) · Springer eBooks · doi:10.1007/978-94-017-6059-1_36 · Cited by 5,146
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
Advances in Large-Margin Classifiers
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/1113.001.0001 · Cited by 1,869
Advances in natural language processing
Hirschberg, J., & Manning, C. D. (2015) · Science · doi:10.1126/science.aaa8685 · Cited by 1,712
Methodology in conducting a systematic review of systematic reviews of healthcare interventions
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-15 · Cited by 1,227 · cited 1×
“However, conducting systematic reviews and meta-analyses is time-consuming and resource-intensive, with some studies estimating up to 72 weeks of skilled labor in the health sciences (Smith et al., 2011).”
Beyond Dichotomies: Competence Viewed as a Continuum
Blömeke, S., Gustafsson, J.-E., & Shavelson, R. J. (2015) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000194 · Cited by 1,205 · cited 1×
“This issue is prevalent in teaching research, which contains a significant number of ambiguous concepts, such as teacher competence or teaching quality (e.g., see Blömeke et al., 2015; Senden et al., 2022).”
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 8×
“There are various machine learning screening tools that are currently available (Burgard & Bittermann, 2023; van de Schoot et al., 2021).”
InfluentialVariables associated with achievement in higher education: A systematic review of meta-analyses.
Schneider, M., & Preckel, F. (2017) · Psychological Bulletin · doi:10.1037/bul0000098 · Cited by 1,083 · cited 1×
“Systematic reviews and meta-analyses are key methods in educational research for advancing policy, research, and practice (e.g., Schneider & Preckel, 2017; van de Schoot et al., 2021; van Huizen & Plantenga, 2018).”
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969 · cited 1×
“In contrast, the conceptual clarity of constructs commonly used in education and educational psychology has been recently criticized (Bringmann et al., 2022; Flake & Fried, 2020; Marsh et al., 2019).”
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638 · cited 1×
“These tools use learning algorithms to acquire knowledge from training data, enabling the identification of specific keywords, phrases, or patterns in newly encountered abstracts and the subsequent prediction of their relevance (Marshall & Wallace, 2019).”
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0782-0 · Cited by 626 · cited 1×
“Previous research suggests that non-experienced reviewers can miss up to 13% of the records in a systematic review, which can significantly impact the results of a meta-analysis (Waffenschmidt et al., 2019).”
Marsh, H. W., Pekrun, R., Parker, P. D., Murayama, K., Guo, J., Dicke, T., & Arens, A. K. (2019) · Journal of Educational Psychology · doi:10.1037/edu0000281 · Cited by 535 · cited 3×
“Previous research suggests that the field of education and educational psychology uses different terms to refer to constructs that are conceptually and empirically similar (e.g., see Marsh et al., 2019).”
Self-citationTheobald, M. (2021) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2021.101976 · Cited by 478 · cited 1×
“For example, in the Theobald (2021) review of the effects of self-regulated learning on university students, it resulted in 46.3% of records for full-text inclusion.”
InfluentialSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 3×
“Time-based approaches involve stopping the review after reviewing a certain number of abstracts (Wallace et al., 2010).”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 343
van Huizen, T., & Plantenga, J. (2018) · Economics of Education Review · doi:10.1016/j.econedurev.2018.08.001 · Cited by 243 · cited 1×
“Systematic reviews and meta-analyses are key methods in educational research for advancing policy, research, and practice (e.g., Schneider & Preckel, 2017; van de Schoot et al., 2021; van Huizen & Plantenga, 2018).”
Endedijk, H. M., Breeman, L. D., Lissa, C. J. V., Hendrickx, M. M. H. G., den Boer, L., & Mainhard, T. (2021) · Review of Educational Research · doi:10.3102/00346543211051428 · Cited by 230
Back to Basics: The Importance of Conceptual Clarification in Psychological Science
Bringmann, L. F., Elmer, T., & Eronen, M. I. (2022) · Current Directions in Psychological Science · doi:10.1177/09637214221096485 · Cited by 221 · cited 1×
“In contrast, the conceptual clarity of constructs commonly used in education and educational psychology has been recently criticized (Bringmann et al., 2022; Flake & Fried, 2020; Marsh et al., 2019).”
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid ThemPreprint
Flake, J. K., & Fried, E. I. (2019) · doi:10.31234/osf.io/hs7wm · Cited by 218
Zhang, Q., Wang, J., & Neitzel, A. J. (2022) · Journal of Youth and Adolescence · doi:10.1007/s10964-022-01684-4 · Cited by 213 · cited 3×
“Similarly, Zhang et al. (2023) used a time-based strategy to stop screening in a systematic review of school-based mental health interventions.”
Children’s Creativity: A Theoretical Framework and Systematic Review
Kupers, E., Lehmann-Wermser, A., McPherson, G., & van Geert, P. (2019) · Review of Educational Research · doi:10.3102/0034654318815707 · Cited by 195
Täschner, J., Dicke, T., Reinhold, S., & Holzberger, D. (2024) · Review of Educational Research · doi:10.3102/00346543231221499 · Cited by 191
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190 · cited 3×
“Whereas several studies have demonstrated the potential of ML-based tools to alleviate reviewer workload and save time in the screening process (Chai et al., 2021; Gates et al., 2019; Hamel et al., 2020; Howard et al., 2020; van de Schoot et al., 2021), there is a noticeable lack of research…”
Rowan, L., Bourke, T., L’Estrange, L., Lunn Brownlee, J., Ryan, M., Walker, S., & Churchward, P. (2021) · Review of Educational Research · doi:10.3102/0034654320979171 · Cited by 176 · cited 1×
“For example, Rowan et al. (2021) systematic review of teacher education and diverse learners focused primarily on the literature on initial teacher education.”
Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0315-4 · Cited by 168 · cited 1×
“Thus, systematic reviews that only rely on human resources are neither efficient nor sustainable, particularly with the rapid increase of scientific information (Shemilt et al., 2016).”
Howard, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157 · cited 5×
“…of ML-based tools to alleviate reviewer workload and save time in the screening process (Chai et al., 2021; Gates et al., 2019; Hamel et al., 2020; Howard et al., 2020; van de Schoot et al., 2021), there is a noticeable lack of research comparing the performance of different learning algorithms…”
InfluentialGates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148 · cited 1×
“Each study is typically reviewed within 30 s (Gates et al., 2018), and, given the increasing number of studies, the abstract screening phase in a systematic review requires a substantial investment of time and effort.”
Multimedia Design for Learning: An Overview of Reviews With Meta-Meta-Analysis
Noetel, M., Griffith, S., Delaney, O., Harris, N. R., Sanders, T., Parker, P., del Pozo Cruz, B., & Lonsdale, C. (2022) · Review of Educational Research · doi:10.3102/00346543211052329 · Cited by 148
Hamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01129-1 · Cited by 140 · cited 1×
“…the potential of ML-based tools to alleviate reviewer workload and save time in the screening process (Chai et al., 2021; Gates et al., 2019; Hamel et al., 2020; Howard et al., 2020; van de Schoot et al., 2021), there is a noticeable lack of research comparing the performance of different…”
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1222-2 · Cited by 133 · cited 3×
“…when using learning algorithms out of the total deemed irrelevant by the authors of the systematic review (Kusa et al., 2023). c. Estimated time savings (ETS): The reduction in working days achieved when using learning algorithms by not having to screen irrelevant abstracts (Gates et al., 2019).”
Natural Language Processing for Online Applications
Jackson, P. E., & Moulinier, I. (2007) · Natural language processing · doi:10.1075/nlp.5 · Cited by 123 · cited 2×
“Based on the Bayes theorem, NB assumes that data features are conditionally independent given their assigned class and contribute equally to the outcome (Gomes et al., 2017; Jackson & Moulinier, 2007).”
Tsou, A. Y., Treadwell, J. R., Erinoff, E., & Schoelles, K. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01324-7 · Cited by 122 · cited 1×
“Thus, the same factors that pose challenges for designing a search with adequate sensitivity and specificity may also pose problems for active learning algorithms and the use of heuristic stopping rules (Tsou et al., 2020).”
Statistical stopping criteria for automated screening in systematic reviews
Callaghan, M. W., & Müller-Hansen, F. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01521-4 · Cited by 121 · cited 6×
“…that the results on the performance of heuristic stopping criteria are highly dependent on dataset characteristics, it is advisable to extend the study to evaluate the effectiveness and generalizability of statistical stopping criteria, such as those proposed by Callaghan and Müller-Hansen (2020).”
InfluentialA Systematic Review of Early Childhood Exclusionary Discipline
Zinsser, K. M., Silver, H. C., Shenberger, E., & Jackson, V. (2022) · Review of Educational Research · doi:10.3102/00346543211070047 · Cited by 117
Hamel, C., Hersi, M., Kelly, S. E., Tricco, A. C., Straus, S., Wells, G., Pham, B., & Hutton, B. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01451-2 · Cited by 104 · cited 2×
“In a mixed-based strategy, researchers decide to stop reviewing after a certain number of abstracts have been reviewed and a certain threshold of consecutive irrelevant records have been found (Hamel et al., 2021; Yu & Menzies, 2019).”
Engineering Quality and Reliability in Technology-Assisted Review
Cormack, G. V., & Grossman, M. R. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911510 · Cited by 98 · cited 3×
“The former estimates the stopping criteria based on the derivation of an estimate of the total number of relevant papers within an initial training set and is considered more reliable than the latter (Cormack & Grossman, 2016; van Haastrecht et al., 2021).”
Mathematical language and mathematical abilities in preschool: A systematic literature review
Turan, E., & Smedt, B. D. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100457 · Cited by 96
Service learning for improving academic success in students in grade K to 12: A systematic review
Filges, T., Dietrichson, J., Viinholt, B. C. A., & Dalgaard, N. T. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1210 · Cited by 92
Refutation Text Facilitates Learning: a Meta-Analysis of Between-Subjects Experiments
Schroeder, N. L., & Kucera, A. C. (2022) · Educational Psychology Review · doi:10.1007/s10648-021-09656-z · Cited by 86
Tarantino, G., Makopoulou, K., & Neville, R. D. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100456 · Cited by 83
FAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80 · cited 4×
“In a mixed-based strategy, researchers decide to stop reviewing after a certain number of abstracts have been reviewed and a certain threshold of consecutive irrelevant records have been found (Hamel et al., 2021; Yu & Menzies, 2019).”
InfluentialSYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing
van Haastrecht, M., Sarhan, I., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.685591 · Cited by 80 · cited 2×
“The former estimates the stopping criteria based on the derivation of an estimate of the total number of relevant papers within an initial training set and is considered more reliable than the latter (Cormack & Grossman, 2016; van Haastrecht et al., 2021).”
Robledo, S., Grisales Aguirre, A. M., Hughes, M., & Eggers, F. (2023) · Journal of Small Business Management · doi:10.1080/00472778.2021.1955125 · Cited by 80 · cited 1×
“…previous research into the performance of various ML-based tools for abstract screening has been limited in sample size, and the software tools evaluated were not open source, thereby hindering the identification of relevant parameters for these methods (Gates et al., 2019; Robledo et al., 2023).”
A Machine Learning Approach for Semi-Automated Search and Selection in Literature Studies
Ros, R., Bjarnason, E., & Runeson, P. (2017) · Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3084226.3084243 · Cited by 80 · cited 1×
“In contrast, data-driven approaches stop after identifying a certain number of consecutive irrelevant abstracts (Ros et al., 2017).”
The Role of Individual Differences in Sourcing: a Systematic Review
Anmarkrud, Ø., Bråten, I., Florit, E., & Masón, L. (2021) · Educational Psychology Review · doi:10.1007/s10648-021-09640-7 · Cited by 79
Filges, T., Sonne‐Schmidt, C. S., & Viinholt, B. C. A. (2018) · Campbell Systematic Reviews · doi:10.4073/csr.2018.10 · Cited by 72
Fong, C. J., Krou, M. R., Johnston-Ashton, K., Hoff, M. A., Lin, S., & Gonzales, C. (2021) · Educational Research Review · doi:10.1016/j.edurev.2021.100407 · Cited by 70
Scherer, R., & Campos, D. G. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100479 · Cited by 63
Self-citationXu, X., Schönrock-Adema, J., Jaarsma, D. A. D. C., Duvivier, R. J., & Bos, N. A. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100474 · Cited by 60
ASReview LAB - A tool for AI-assisted systematic reviews
developers, A. L. (2026) · Zenodo · doi:10.5281/zenodo.3345592 · Cited by 59
Senden, B., Nilsen, T., & Blömeke, S. (2022) · Ways of Analyzing Teaching Quality · doi:10.18261/9788215045054-2021-05 · Cited by 56
The dilemma and countermeasures of educational data ethics in the age of intelligence
Guan, X., Feng, X., & Islam, A. Y. M. A. (2023) · Humanities and Social Sciences Communications · doi:10.1057/s41599-023-01633-x · Cited by 55 · cited 3×
“Hence, more and more researchers are turning to learning algorithms to automate this screening phase (e.g., Guan et al., 2023; Huang et al., 2022; Scherer & Campos, 2022; Zhang et al., 2023).”
Roberts, G. J., Dumas, D., McNeish, D., & Coté, B. (2021) · Review of Educational Research · doi:10.3102/00346543211051423 · Cited by 55
Active learning for screening prioritization in systematic reviews - A simulation study
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020) · doi:10.31219/osf.io/w6qbg · Cited by 51 · cited 2×
“An attempt to 19 Page 6 of 30 address the previous research gaps was presented by Ferdinands et al. (2020).”
Networks in Education: A Travelogue Through Five Decades
Saqr, M., Poquet, O., & López‐Pernas, S. (2022) · IEEE Access · doi:10.1109/access.2022.3159674 · Cited by 42
Lesperance, K., Hofer, S. I., Retelsdorf, J., & Holzberger, D. (2022) · British Journal of Educational Psychology · doi:10.1111/bjep.12512 · Cited by 42
König, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1762 · Cited by 39
Self-citationNeri, N. C., & Retelsdorf, J. (2022) · Educational Research Review · doi:10.1016/j.edurev.2022.100460 · Cited by 36
A comparative approach to email classification using Naive Bayes classifier and hidden Markov model
Gomes, S. R., Saroar, S. G., Mosfaiul, M., et al. (2017) · doi:10.1109/icaee.2017.8255404 · Cited by 35
König, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M. (2023) · doi:10.31234/osf.io/ybu3w · Cited by 33
Self-citationUsing natural language processing and machine learning to replace human content coders.
Wang, Y., Tian, J., Yazar, Y., et al. (2022) · Psychological Methods · doi:10.1037/met0000518 · Cited by 31 · cited 3×
“In contrast, model training involves fitting learning algorithms to the engineered dataset (Wang et al., 2022).”
InfluentialBurgard, T., & Bittermann, A. (2023) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000509 · Cited by 29 · cited 6×
“Additionally, the current literature lacks comprehensive comparative studies on the performance of learning algorithms when applied to real-world data (Burgard & Bittermann, 2023), and the impact of data characteristics on model performance is poorly understood (Ferdi-nands et al., 2020).”
InfluentialHuang, Y., Procházková, M., Lu, J., Riad, A., & Macek, P. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.871795 · Cited by 21 · cited 2×
“Hence, more and more researchers are turning to learning algorithms to automate this screening phase (e.g., Guan et al., 2023; Huang et al., 2022; Scherer & Campos, 2022; Zhang et al., 2023).”
Kusa, W., Lipani, A., Knoth, P., & Hanbury, A. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200193 · Cited by 21 · cited 1×
“…would not require screening by reviewers when using learning algorithms out of the total deemed irrelevant by the authors of the systematic review (Kusa et al., 2023). c. Estimated time savings (ETS): The reduction in working days achieved when using learning algorithms by not having to screen…”
EDISON: Feature Extraction for NLP, Simplified
Sammons, M., Christodoulopoulos, C., Kordjamshidi, P., Khashabi, D., Srikumar, V., Vijayakumar, P., Bokhari, M., Wu, X., & Roth, D. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/36b8cp6hbxke · Cited by 20
Shared Book Reading for Spanish-Speaking Emergent Bilinguals: A Review of Experimental Studies
Pico, D. L., & Woods, C. (2022) · Review of Educational Research · doi:10.3102/00346543221095112 · Cited by 17
Täschner, J., Dicke, T., Reinhold, S., & Holzberger, D. (2023) · doi:10.31234/osf.io/cds45 · Cited by 11
Gestaltung und Evaluation von webbasierten Lehrkräftefortbildungen: Ein Systematic Review
Capparozza, M., Fröhlich, N., Dehmel, A., & Fauth, B. (2023) · Edition ZfE · doi:10.1007/978-3-658-37895-0_15 · Cited by 10
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · doi:10.23970/ahrqepcmethmachineperformance · Cited by 8
Filges, T., Dietrichson, J., Viinholt, B. C. A., & Dalgaard, N. T. (2021) · Campbell Systematic Reviews · doi:10.1002/cl2.1157 · Cited by 8
Teijema, J. J., van de Schoot, R., Ferdinands, G., Lombaers, P., & Bruin, J. D. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7550649 · Cited by 8
Systematic reviews as object to study relevance assessment processes
Jäger-Dengler-Harles, I., Heck, T., & Rittberger, M. (2020) · doi:10.47989/irisic2024 · Cited by 5
Scherer, R., & Campos, D. (2021) · doi:10.31234/osf.io/82ucv · Cited by 4
Self-citationBackfisch, I., Schneider, J. M., Lachner, A., et al. (2020) · doi:10.23668/psycharchives.4226 · Cited by 3
Self-citationStatistical Stopping Criteria for Automated Screening in Systematic Reviews
Callaghan, M., & Müller-Hansen, F. (2020) · doi:10.21203/rs.2.18218/v2
Zhang, Q. (2023) · Proceedings of the 2023 AERA Annual Meeting · doi:10.3102/2001548
Multimedia design for learning: An overview of reviews with meta-meta-analysis
Noetel, M., Griffith, S., Delaney, O., Harris, N. R., Sanders, T., Parker, P. D., Cruz, B. D. P., & Lonsdale, C. (2021) · doi:10.31234/osf.io/pynzr
Research works printed but not linked: 3
- Fütterer, T., Tschönhens, F., Scherer, R., Dickhäuser, O., & Ruiz-Primo, M. A. (2023). Teachers’ self-assessment ability—A systematic literature review and meta-analysis. [Manuscript in preparation]. Faculty of Economic and Social Sciences. University of Tübingen.
- Veletić, J., Rodriguez-Mejia, A. M., & Olsen, R. V. (2023). A systematic literature review of the Teaching and Learning International Survey (TALIS) research. Faculty of Educational Sciences. University of Oslo
- World Health Organization. (2019). The ICD-10 classification of mental and behavioural disorders: Diagnostic criteria for research. World Health Organization.
Cited by
Enhancing peer assessment with artificial intelligence
Topping, K. J., Gehringer, E. F., Khosravi, H., Gudipati, S., Jadhav, K., & Susarla, S. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00501-1 · Cited by 104
Bourke, M., Wang, H., McNaughton, S. A., Thomas, G., Firth, J., Trott, M., & Cairney, J. (2025) · Clinical Psychology Review · doi:10.1016/j.cpr.2025.102585 · Cited by 75
The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence
Slattery, P., Saeri, A. K., Grundy, E. A. C., Graham, J., Noetel, M., Uuk, R., Dao, J., Pour, S., Casper, S., & Thompson, N. (2026) · Patterns · doi:10.1016/j.patter.2026.101517 · Cited by 255
“Based on a review of relevant literature (van de Schoot et al 42 ; e.g., Campos et al. 44 ), we use the Oracle model with the following setup: a fully connected neural network (two hidden layers) model as the classifier and sBert as the feature extractor, maximum as the query strategy, and dynamic resampling (double)…”
InfluentialKönig, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1762 · Cited by 39
“16,19,26,38,39 Consequently, performance variations may arise when using AI-AS for abstract collections from other research domains.”
InfluentialSelf-citationASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts
de Bruin, J., Lombaers, P., Kaandorp, C., Teijema, J., van der Kuil, T., Yazan, B., Dong, A., & van de Schoot, R. (2025) · Patterns · doi:10.1016/j.patter.2025.101318 · Cited by 27
“32,39,43 There is also the global stopping condition next to the cycle’s stopping condition, for example, when the end of data is reached or if no more agents are available.”
Metadata
- Authors
- Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R.
- Affiliations
- University of Oslo (NO)
- University of Tübingen (DE)
- University of Copenhagen (DK)
- Helmut Schmidt University (DE)
- Corresponding author
- Campos, Diego G.
- ORCID
- Venue
- Educational Psychology Review
- Published
- 2024-02-08
- Volume
- 36
- Issue
- 1
- ISSN
- 1040-726X (ISSN-L)
- 1573-336X
- OpenAlex
- W4391653771
- Cited by
- 75
- References
- 74
- Citations by year
- 2026: 33 · 2025: 32 · 2024: 10
- FWCI
- 9.43
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Educational Psychology Review (publishedVersion, cc-by)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Funders
- University of Oslo
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- systematic review screening, abstract screening, random forest, BERT, educational research, educational psychology, sensitivity and specificity, time savings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.