Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 10 of the 24 sources its claims cite were checked for support.
Abstract
Traditional systematic reviews, despite their high-quality evidence, are labor-intensive and error-prone, especially during the abstract screening phase. This paper investigates the application of machine learning-assisted systematic reviewing in the context of Learning Analytics (LA) in higher education. This study evaluates two approaches—ASReview, an active traditional machine learning tool, and GPT-4o, a large language model—to automate this process. By comparing key performance metrics such as sensitivity, specificity, accuracy, precision, and F1-score, we assess the effectiveness of these tools against traditional manual methods. Our findings demonstrate the potential of machine learning to enhance the efficiency and accuracy of systematic reviews in learning analytics.
Overview
CoChat Summary
TL;DR
This case study compares ASReview (traditional active machine learning) and GPT-4o for abstract screening in a systematic review of learning analytics in higher education, using manual screening as the benchmark. ASReview achieved very high sensitivity but low precision and substantial manual workload, while GPT-4o achieved more balanced specificity, accuracy, and F1-score.
Key findings
- ASReview at WSS@95% achieved mean sensitivity 0.950 (95% CI 0.946–0.954), specificity 0.346–0.409, precision 0.204–0.233, accuracy 0.444–0.488, and F1-score 0.337–0.373; reaching 95% recall required screening approximately 4,833–5,585 records, about 68–80% of the 7,020 total records.
- ASReview workload reduction varied by classifier: Logistic Regression saved 31.12% (95% CI 18.33–43.91), Naïve Bayes 27.26% (95% CI 13.94–40.59), and Random Forest 20.40% (95% CI 9.12–31.68) at WSS@95%.
- GPT-4o over ten independent runs achieved mean sensitivity 0.795 (95% CI 0.790–0.800), specificity 0.688 (95% CI 0.684–0.692), accuracy 0.704 (95% CI 0.701–0.707), precision 0.314 (95% CI 0.311–0.317), and F1-score 0.449 (95% CI 0.445–0.453), with mean confusion counts TP = 847, FP = 1,870, TN = 4,080, FN = 219.
- The paper reports that GPT-4o had higher specificity, accuracy, and F1-score than ASReview, while ASReview maximized recall; a formal paired statistical comparison such as McNemar’s test could not be conducted because results came from separate simulation runs rather than paired record-level predictions.
Methods
- Design: a case study using a previously completed systematic review on learning analytics in higher education; manual title/abstract screening decisions served as ground-truth labels for evaluating ASReview and GPT-4o.
- Data: searches were limited to 2011–2023 and conducted in ERIC, Education Source, APA PsycInfo, Web of Science Core Collection, IEEExplore, and Applied Science and Technology Source Ultimate; after removing 1,603 duplicates, 7,016 unique references were screened, leaving 1,086 articles for full-text screening, then 173 for coding, and finally 159 articles generating 161 independent studies.
- Analysis: ASReview v1.6.2 was tested with Naïve Bayes, Random Forest, and Logistic Regression using 10 Monte Carlo runs per classifier with the same prior knowledge set of five relevant and five irrelevant records; GPT-4o was tested over 10 independent runs with identical prompts and parameters (temperature 0.50, max tokens 1, top-p 1.00, frequency and presence penalties 0). Performance was assessed with sensitivity, specificity, precision, accuracy, and F1-score derived from confusion matrices, with mean values and 95% confidence intervals.
Limitations
- The paper states limitations including single-case focus on higher education, evaluation of only two tools due to resource constraints, only ten GPT-4o trials, no paired record-level decisions preventing McNemar’s test, no ASReview learning-curve analyses, no GPT-4o prompt-sensitivity analysis, no detailed error analysis of failure cases, no testing of alternative tools, and no validation across multiple datasets or domains, constraining generalizability.
- The paper also states that effectiveness depends on data quality and input biases, ChatGPT output is sensitive to prompt phrasing with no agreement on best prompt characteristics, tools require technical expertise and may incur costs, LLMs can inherit biases such as ethnic or gender biases, and ML models can produce unfair outcomes due to data, development, and interaction factors.
- Evident from the text: ASReview and GPT-4o results were obtained from separate simulation runs rather than paired record-level predictions on the same dataset, so performance comparisons relied on run-level means and 95% confidence intervals rather than formal paired statistical tests.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 0 Not valid · 10 Not checked · 75 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Traditional systematic reviews, despite their high-quality evidence, are labor-intensive and error-prone, especially during the abstract screening phase.”
- Also stated in: § 1. Introduction · plain
“Systematic reviews and meta-analyses, although providing high-quality evidence, are often time-consuming and labor costly, particularly due to the manual screening of titles, abstracts, and full texts (Chai et al., 2021).”
§ Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics > Abstract
What the sources say
- Partly supportedcites Chai et al., 2021 sourceExperimental — accuracy not yet measured
The Background describes systematic reviews as high-quality evidence with significant time and economic demands, and identifies title and abstract screening as the most time-consuming stage. The Discussion’s Strengths and limitations section notes a risk of missing eligible articles when using Research Screener, but it does not establish that traditional reviews or manual abstract screening are especially error-prone. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1. Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Despite these limitations, prior studies and our findings suggest that GPT outperforms traditional active learning methods in terms of time efficiency (Alshami et al., 2023).”
§ Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics > 3.3.2. Accuracy, F1-Score and Overall Performance
What the sources say
- Partly supportedcites Alshami et al., 2023 sourceExperimental — accuracy not yet measured
Section 4.1 reports that ChatGPT reduced time and effort in systematic-review tasks. The paper compares its classification results with expert assessments, but the passages do not compare its time efficiency with traditional active-learning methods. That comparative part of the claim is unsupported. (LLM, conf=0.86)
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specificity, on the other hand, reflects the model's ability to correctly exclude irrelevant articles, which in this case is 68.80%.”
§ Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics > 3.2. Large Language Model
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In ASReview, recall was fixed at 0.95 for the WSS@95 comparison, yielding a mean sensitivity of 0.950 (95% CI 0.946 to 0.954) across classifiers (Table 3).”
§ Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics > 3.3.1. Sensitivity, Specificity, and Precision
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, specificity remained relatively low (0.346-0.409), meaning that only about 37% of irrelevant records were correctly excluded.”
§ Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics > 3.1. Traditional Machine Learning
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
18 more claims
Create a free account to see all 93 claims and every source verdict.
- Borderline: one of the tests for a substantial claim is unclear. 1
- It may state two separate claims as one. 50
- Probably not a substantial claim. 31
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics > 2.1. Data
“After removing 1603 duplicates, 7016 unique references were screened.”
§ Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics > 3.3.1. Sensitivity, Specificity, and Precision
“Achieving this recall required screening approximately 4833 to 5585 records, which represents about 68 to 80% of the 7020 total records”
Citations
Cites
Evaluation metrics and statistical tests for machine learning
Rainio, O., Teuho, J., & Klén, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-56706-x · Cited by 1,338
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Ethical and Bias Considerations in Artificial Intelligence/Machine Learning
Hanna, M. G., Pantanowitz, L., Jackson, B., Palmer, O., Visweswaran, S., Pantanowitz, J., Deebajah, M., & Rashidi, H. H. (2025) · Modern Pathology · doi:10.1016/j.modpat.2024.100686 · Cited by 535
Pranckevičius, T., & Marcinkevičius, V. (2017) · Baltic Journal of Modern Computing · doi:10.22364/bjmc.2017.5.2.05 · Cited by 307
Automated Paper Screening for Clinical Reviews Using Large Language Models
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00844 · Cited by 230
Automated paper screening for clinical reviews using large language models: Data analysis study
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2024) · Journal of Medical Internet Research · doi:10.2196/48996 · Cited by 205
Alshami, A., Elsayed, M., Ali, E., Eltoukhy, A. E. E., & Zayed, T. (2023) · Systems · doi:10.3390/systems11070351 · Cited by 202
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
Bannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185
On the confusion matrix in credit scoring and its analytical properties
Zeng, G. (2020) · Communications in Statistics - Theory and Methods · doi:10.1080/03610926.2019.1568485 · Cited by 157
Research works printed but not linked: 2
- Spedener, L. (2025, November 01). Towards performance comparability: An implementation of new metrics into the ASReview active learning screening prioritization software for systematic literature reviews [Unpublished Master’s Thesis, Utrecht University], Available online: https://studenttheses.uu.nl/handle/20.500.12932/45187.
- Zhu, 2010, Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations, NESUG Proceeding: Health Care and Life Sciences, Vol. 19, 67
Cited by
Scoping Review of Pre-Consumption Food Loss in the US Supply Chain: Factors, Impacts, and Solutions
Ma, S., Kandi, L. P., Xu, Z., Lu, P., & Dooley, K. E. (2026) · Foods · doi:10.3390/foods15040775 · Cited by 2
“For example, Xu et al. [51] explored machine learning methods and large language models for assisting the title and abstract stage in systematic reviews.”
Self-citation
Metadata
- Authors
- Xu, Z., Zhuang, X., & Ma, S.
- Affiliations
- Texas A&M University (US)
- North Dakota State University (US)
- Corresponding author
- Xu, Zhihong
- Venue
- Education Sciences
- Publisher
- MDPI AG
- Published
- 2025-11-05
- Volume
- 15
- Issue
- 11
- Pages
- 1488
- ISSN
- 2227-7102 (ISSN-L)
- OpenAlex
- W4415923729
- Cited by
- 2
- References
- 26
- Citations by year
- 2026: 2
- FWCI
- 0.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Education Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Homeland Security (18STCBT00001)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- learning analytics, abstract screening, ASReview, GPT-4o, large language models, literature review automation, higher education, sensitivity and specificity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.