CoChat

Generative AI in Higher Education: Evidence Synthesis of Applied Learning Outcomes

Open access
Muhammad Nazirul Mubin Bin Abdul Moen, Alfa Rohmatin, Mia Siti Nurazizah, Mehwish Kaleem · 2026 · International Journal of Applied Research and Innovation

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index; 2 of the 35 sources its claims cite were checked for support.

Abstract

Generative artificial intelligence has spread through higher education faster than institutions have produced stable evidence on learning effects. This study synthesized recent empirical reviews, meta-analyses, and applied higher-education research to identify where generative artificial intelligence improved learning and where outcomes remained contingent on instructional design. A structured evidence-synthesis method compared five quantitative or systematic anchor reviews with complementary studies on assessment, feedback, academic integrity, and learner regulation. The anchor literature ranged from 34 to 103 included studies, while a 2026 three-level meta-analysis reported an overall learning effect of g = 0.499 across 36 studies, 132 effect sizes, and 7,229 participants. Larger effects were reported for collaborative learning and cognitive or creative outcomes, while other reviews showed positive achievement and motivation effects but inconsistent self-efficacy findings. The synthesis concluded that generative artificial intelligence produced stronger outcomes when embedded in guided, task-specific pedagogy rather than used as an unrestricted answer generator. Assessment redesign, verification routines, and learner metacognition remained necessary implementation conditions

Overview

CoChat Summary

TL;DR

This structured evidence synthesis reports that generative AI in higher education is associated with positive but conditional learning effects, anchored by an overall g = 0.499 in a three-level meta-analysis of 36 studies, 132 effect sizes, and 7,229 participants. The synthesis concludes benefits are stronger when generative AI is embedded in guided, task-specific pedagogy requiring reasoning, verification, feedback, and transfer rather than used as an unrestricted answer generator.

Key findings

  • Fan et al. (2026) reported an overall learning effect of g = 0.499 across 36 studies, 132 effect sizes, and 7,229 participants; domain effects were g = 0.504 for higher-order thinking, g = 0.669 for cognitive or creative outcomes, g = 1.026 for collaborative learning, and g = 0.633 for blended learning.
  • Liu et al. (2025) reported effects of 0.857 for learning achievement and 0.803 for motivation across 49 articles, while Deng et al. (2025) reported positive effects across several outcome families but no statistically significant improvement in self-efficacy.
  • Anchor reviews ranged from 34 included studies in Weng et al. (2024) to 103 empirical studies in Hawi and Samaha (2026), with Baig and Yadegaridehkordi (2024) reviewing 57 studies and Liu et al. (2025) synthesizing 49 articles.
  • The synthesis concluded that generative AI produced stronger outcomes when embedded in guided, task-specific pedagogy, with assessment redesign, verification routines, and learner metacognition as necessary implementation conditions.

Methods

  • Structured evidence-synthesis design; the paper states five anchor sources were selected, then names Weng et al. (2024), Baig and Yadegaridehkordi (2024), Deng et al. (2025), Liu et al. (2025), Fan et al. (2026), and Hawi and Samaha (2026), complemented by peer-reviewed studies on assessment, academic integrity, language learning, and learner perceptions.
  • Eligible evidence had to discuss generative AI or large language models in educational settings and provide an identifiable implication for learning, teaching, assessment, or implementation; sources focused solely on technical model architecture without an educational outcome were excluded.
  • Data extraction recorded publication year, review type, number of included studies when reported, participant counts where available, outcome domains, effect estimates, and implementation moderators; effects were not pooled again but preserved as reported and compared descriptively, with interpretive coding of four mechanisms and risks.

Limitations

  • Paper states: review overlap means some primary studies may contribute to more than one published synthesis; intervention durations are often short; rapidly changing model capabilities can make evidence temporally fragile; effect sizes from different meta-analyses cannot be directly pooled without common primary-study data and harmonized coding; longitudinal evidence on retention, transfer, and independent competence remains thinner than evidence on immediate task performance.
  • Paper states: the method does not estimate publication bias across the entire primary literature and cannot resolve dependence where the same primary studies appear in multiple reviews; it is deliberately structured rather than claimed as an exhaustive PRISMA systematic review.
  • Evident in text: the methods describe “five anchor sources” but then name six sources; the review does not conduct de novo screening of every primary study.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.65310/x5zzye23

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY-SA · From the PDF
Figure 1 Scale of Selected Generative-AI Evidence Syntheses
Figure 1 Scale of Selected Generative-AI Evidence SynthesesCC-BY-SA · From the PDF
Figure 2 Selected Reported Learning Effects
Figure 2 Selected Reported Learning EffectsCC-BY-SA · From the PDF

Verification

Claims the paper makes

2 Valid · 1 Not valid · 28 Not checked · 71 Own results and interpretations

41 more claims

Create a free account to see all 102 claims and to verify the 31 references still open.

See all claims — free account
  • It may state two separate claims as one. 26
  • Probably not a substantial claim. 39

Create a free account to see these claims.

Citations

Cites

Cited by

Still being checked.

Metadata

Authors
Moen, M. N. M. B. A., Rohmatin, A., Nurazizah, M. S., & Kaleem, M.
Affiliations
International Islamic University Malaysia (MY)
Venue
International Journal of Applied Research and Innovation
Published
2026-10-20
Volume
1
Issue
2
Pages
159-169
ISSN
  • 3124-3665 (ISSN-L)
  • 3124-3630
OpenAlex
W7223092347
Cited by
0
References
26
FWCI
0.00
Type
journal article
Language
en
Open access
Locations
International Journal of Applied Research and Innovation (publishedVersion, cc-by-sa)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
generative AI, higher education, learning outcomes, instructional design, cooperative learning, academic integrity, assessment redesign, learner metacognition, student motivation, self-efficacy, feedback, creative outcomes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.