CoChat

Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics

Open access
Xu, Zhihong, Zhuang, Xiting, Ma, Shuai · 2025 · Education Sciences

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 10 of the 24 sources its claims cite were checked for support.

Abstract

Traditional systematic reviews, despite their high-quality evidence, are labor-intensive and error-prone, especially during the abstract screening phase. This paper investigates the application of machine learning-assisted systematic reviewing in the context of Learning Analytics (LA) in higher education. This study evaluates two approaches—ASReview, an active traditional machine learning tool, and GPT-4o, a large language model—to automate this process. By comparing key performance metrics such as sensitivity, specificity, accuracy, precision, and F1-score, we assess the effectiveness of these tools against traditional manual methods. Our findings demonstrate the potential of machine learning to enhance the efficiency and accuracy of systematic reviews in learning analytics.

Overview

CoChat Summary

TL;DR

This case study compares ASReview (traditional active machine learning) and GPT-4o for abstract screening in a systematic review of learning analytics in higher education, using manual screening as the benchmark. ASReview achieved very high sensitivity but low precision and substantial manual workload, while GPT-4o achieved more balanced specificity, accuracy, and F1-score.

Key findings

  • ASReview at WSS@95% achieved mean sensitivity 0.950 (95% CI 0.946–0.954), specificity 0.346–0.409, precision 0.204–0.233, accuracy 0.444–0.488, and F1-score 0.337–0.373; reaching 95% recall required screening approximately 4,833–5,585 records, about 68–80% of the 7,020 total records.
  • ASReview workload reduction varied by classifier: Logistic Regression saved 31.12% (95% CI 18.33–43.91), Naïve Bayes 27.26% (95% CI 13.94–40.59), and Random Forest 20.40% (95% CI 9.12–31.68) at WSS@95%.
  • GPT-4o over ten independent runs achieved mean sensitivity 0.795 (95% CI 0.790–0.800), specificity 0.688 (95% CI 0.684–0.692), accuracy 0.704 (95% CI 0.701–0.707), precision 0.314 (95% CI 0.311–0.317), and F1-score 0.449 (95% CI 0.445–0.453), with mean confusion counts TP = 847, FP = 1,870, TN = 4,080, FN = 219.
  • The paper reports that GPT-4o had higher specificity, accuracy, and F1-score than ASReview, while ASReview maximized recall; a formal paired statistical comparison such as McNemar’s test could not be conducted because results came from separate simulation runs rather than paired record-level predictions.

Methods

  • Design: a case study using a previously completed systematic review on learning analytics in higher education; manual title/abstract screening decisions served as ground-truth labels for evaluating ASReview and GPT-4o.
  • Data: searches were limited to 2011–2023 and conducted in ERIC, Education Source, APA PsycInfo, Web of Science Core Collection, IEEExplore, and Applied Science and Technology Source Ultimate; after removing 1,603 duplicates, 7,016 unique references were screened, leaving 1,086 articles for full-text screening, then 173 for coding, and finally 159 articles generating 161 independent studies.
  • Analysis: ASReview v1.6.2 was tested with Naïve Bayes, Random Forest, and Logistic Regression using 10 Monte Carlo runs per classifier with the same prior knowledge set of five relevant and five irrelevant records; GPT-4o was tested over 10 independent runs with identical prompts and parameters (temperature 0.50, max tokens 1, top-p 1.00, frequency and presence penalties 0). Performance was assessed with sensitivity, specificity, precision, accuracy, and F1-score derived from confusion matrices, with mean values and 95% confidence intervals.

Limitations

  • The paper states limitations including single-case focus on higher education, evaluation of only two tools due to resource constraints, only ten GPT-4o trials, no paired record-level decisions preventing McNemar’s test, no ASReview learning-curve analyses, no GPT-4o prompt-sensitivity analysis, no detailed error analysis of failure cases, no testing of alternative tools, and no validation across multiple datasets or domains, constraining generalizability.
  • The paper also states that effectiveness depends on data quality and input biases, ChatGPT output is sensitive to prompt phrasing with no agreement on best prompt characteristics, tools require technical expertise and may incur costs, LLMs can inherit biases such as ethnic or gender biases, and ML models can produce unfair outcomes due to data, development, and interaction factors.
  • Evident from the text: ASReview and GPT-4o results were obtained from separate simulation runs rather than paired record-level predictions on the same dataset, so performance comparisons relied on run-level means and 95% confidence intervals rather than formal paired statistical tests.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/educsci15111488

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

8 Valid · 0 Not valid · 10 Not checked · 75 Own results and interpretations

18 more claims

Create a free account to see all 93 claims and every source verdict.

See all claims — free account
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • It may state two separate claims as one. 50
  • Probably not a substantial claim. 31

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 2

  1. Spedener, L. (2025, November 01). Towards performance comparability: An implementation of new metrics into the ASReview active learning screening prioritization software for systematic literature reviews [Unpublished Master’s Thesis, Utrecht University], Available online: https://studenttheses.uu.nl/handle/20.500.12932/45187.
  2. Zhu, 2010, Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations, NESUG Proceeding: Health Care and Life Sciences, Vol. 19, 67

Cited by

See all 2

Metadata

Authors
Xu, Z., Zhuang, X., & Ma, S.
Affiliations
  • Texas A&M University (US)
  • North Dakota State University (US)
Corresponding author
Xu, Zhihong
Venue
Education Sciences
Publisher
MDPI AG
Published
2025-11-05
Volume
15
Issue
11
Pages
1488
ISSN
2227-7102 (ISSN-L)
OpenAlex
W4415923729
Cited by
2
References
26
Citations by year
2026: 2
FWCI
0.21
Type
journal article
Language
en
Open access
Locations
  • Education Sciences (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
U.S. Department of Homeland Security (18STCBT00001)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
learning analytics, abstract screening, ASReview, GPT-4o, large language models, literature review automation, higher education, sensitivity and specificity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.