Generative AI in Higher Education: Evidence Synthesis of Applied Learning Outcomes
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index; 2 of the 35 sources its claims cite were checked for support.
Abstract
Generative artificial intelligence has spread through higher education faster than institutions have produced stable evidence on learning effects. This study synthesized recent empirical reviews, meta-analyses, and applied higher-education research to identify where generative artificial intelligence improved learning and where outcomes remained contingent on instructional design. A structured evidence-synthesis method compared five quantitative or systematic anchor reviews with complementary studies on assessment, feedback, academic integrity, and learner regulation. The anchor literature ranged from 34 to 103 included studies, while a 2026 three-level meta-analysis reported an overall learning effect of g = 0.499 across 36 studies, 132 effect sizes, and 7,229 participants. Larger effects were reported for collaborative learning and cognitive or creative outcomes, while other reviews showed positive achievement and motivation effects but inconsistent self-efficacy findings. The synthesis concluded that generative artificial intelligence produced stronger outcomes when embedded in guided, task-specific pedagogy rather than used as an unrestricted answer generator. Assessment redesign, verification routines, and learner metacognition remained necessary implementation conditions
Overview
CoChat Summary
TL;DR
This structured evidence synthesis reports that generative AI in higher education is associated with positive but conditional learning effects, anchored by an overall g = 0.499 in a three-level meta-analysis of 36 studies, 132 effect sizes, and 7,229 participants. The synthesis concludes benefits are stronger when generative AI is embedded in guided, task-specific pedagogy requiring reasoning, verification, feedback, and transfer rather than used as an unrestricted answer generator.
Key findings
- Fan et al. (2026) reported an overall learning effect of g = 0.499 across 36 studies, 132 effect sizes, and 7,229 participants; domain effects were g = 0.504 for higher-order thinking, g = 0.669 for cognitive or creative outcomes, g = 1.026 for collaborative learning, and g = 0.633 for blended learning.
- Liu et al. (2025) reported effects of 0.857 for learning achievement and 0.803 for motivation across 49 articles, while Deng et al. (2025) reported positive effects across several outcome families but no statistically significant improvement in self-efficacy.
- Anchor reviews ranged from 34 included studies in Weng et al. (2024) to 103 empirical studies in Hawi and Samaha (2026), with Baig and Yadegaridehkordi (2024) reviewing 57 studies and Liu et al. (2025) synthesizing 49 articles.
- The synthesis concluded that generative AI produced stronger outcomes when embedded in guided, task-specific pedagogy, with assessment redesign, verification routines, and learner metacognition as necessary implementation conditions.
Methods
- Structured evidence-synthesis design; the paper states five anchor sources were selected, then names Weng et al. (2024), Baig and Yadegaridehkordi (2024), Deng et al. (2025), Liu et al. (2025), Fan et al. (2026), and Hawi and Samaha (2026), complemented by peer-reviewed studies on assessment, academic integrity, language learning, and learner perceptions.
- Eligible evidence had to discuss generative AI or large language models in educational settings and provide an identifiable implication for learning, teaching, assessment, or implementation; sources focused solely on technical model architecture without an educational outcome were excluded.
- Data extraction recorded publication year, review type, number of included studies when reported, participant counts where available, outcome domains, effect estimates, and implementation moderators; effects were not pooled again but preserved as reported and compared descriptively, with interpretive coding of four mechanisms and risks.
Limitations
- Paper states: review overlap means some primary studies may contribute to more than one published synthesis; intervention durations are often short; rapidly changing model capabilities can make evidence temporally fragile; effect sizes from different meta-analyses cannot be directly pooled without common primary-study data and harmonized coding; longitudinal evidence on retention, transfer, and independent competence remains thinner than evidence on immediate task performance.
- Paper states: the method does not estimate publication bias across the entire primary literature and cannot resolve dependence where the same primary studies appear in multiple reviews; it is deliberately structured rather than claimed as an exhaustive PRISMA systematic review.
- Evident in text: the methods describe “five anchor sources” but then name six sources; the review does not conduct de novo screening of every primary study.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
2 Valid · 1 Not valid · 28 Not checked · 71 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Liu et al. (2025) reported effects of 0.857 for learning achievement and 0.803 for motivation”
- Also stated in: § Figure 1. Scale of Selected Generative-AI Evidence Syntheses · plain
“Liu et al. (2025) reported large positive effects for learning achievement (0.857) and motivation (0.803) across 49 articles.”
§ Generative AI in Higher Education: Evidence Synthesis of Applied Learning Outcomes > RESEARCH METHODS
What the sources say
- Supportedcites Liu et al. (2025) source
Table 4 ("The Overall Effect of GenAI Application on Learning Achievement and Motivation," p. of Limitations/tables section) lists random-effects effect sizes g = 0.857 for learning achievement (95% CI [0.601, 1.113]) and g = 0.803 for learning motivation (95% CI [0.445, 1.161]), N = 49 and 30 respectively. Table 6 (Trim and Fill) repeats the observed values 0.857 and 0.803, and the overall-effect narrative ("Analysis of the Overall Effect…") describes these as large positive effects. The attributed magnitudes therefore match our copy exactly. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Figure 1. Scale of Selected Generative-AI Evidence Syntheses · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Liu et al. (2025) reported effects of 0.857 for learning achievement and 0.803 for motivation”
- Also stated in: § Figure 1. Scale of Selected Generative-AI Evidence Syntheses · plain
“Liu et al. (2025) reported large positive effects for learning achievement (0.857) and motivation (0.803) across 49 articles.”
§ Generative AI in Higher Education: Evidence Synthesis of Applied Learning Outcomes > RESEARCH METHODS
What the sources say
- Supportedcites Liu et al. (2025) source
Table 4 ('The Overall Effect of GenAI Application on Learning Achievement and Motivation', Limitations and Future Studies section, p. ~14) lists Hedges' g = 0.857 (95% CI [0.601, 1.113], SE 0.128, N = 49, random model) for learning achievement and g = 0.803 (95% CI [0.445, 1.161], SE 0.180, N = 30, random model) for learning motivation. The same two values appear as the observed point estimates in Table 6 (trim and fill), confirming they are the reported overall effects rather than adjusted ones. Both figures in the citing sentence match the source. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Figure 1. Scale of Selected Generative-AI Evidence Syntheses · plain
In the paper
“a 2026 three-level meta-analysis reported an overall learning effect of g = 0.499 across 36 studies, 132 effect sizes, and 7,229 participants”
- Also stated in: § INTRODUCTION · plain
“Fan et al. (2026) analyzed 36 studies, 132 effect sizes, and 7,229 participants using a three-level model, reporting an overall effect of g = 0.499”
- Also stated in: § RESULTS AND DISCUSSION Magnitude and distribution of reported learning effects · plain
“Their three-level meta-analysis covered 36 studies, 132 effect sizes, and 7,229 participants and produced an overall effect of g = 0.499.”
§ Generative AI in Higher Education: Evidence Synthesis of Applied Learning Outcomes > Abstract
What the sources say
- Not checked yetcites Fan et al. (2026) source
plainn = 7,229- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § INTRODUCTION · plain
In the paper
“Fan et al. (2026) reported g = 0.504 for higher-order thinking”
- Also stated in: § RESEARCH METHODS · plain
“g = 0.504 for higher-order thinking”
§ Generative AI in Higher Education: Evidence Synthesis of Applied Learning Outcomes > Figure 1. Scale of Selected Generative-AI Evidence Syntheses
What the sources say
- Not checked yetcites Fan et al. (2026) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § RESEARCH METHODS · plain
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Study counts are not directly comparable because eligibility criteria differ”
§ Generative AI in Higher Education: Evidence Synthesis of Applied Learning Outcomes > RESULTS AND DISCUSSION Magnitude and distribution of reported learning effects
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
41 more claims
Create a free account to see all 102 claims and to verify the 31 references still open.
- It may state two separate claims as one. 26
- Probably not a substantial claim. 39
Citations
Cites
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., et al. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102642 · Cited by 4,584
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
Chatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087
Students’ voices on generative AI: perceptions, benefits, and challenges in higher education
Chan, C. K. Y., & Hu, W. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00411-8 · Cited by 2,301
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024
What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00237-x · Cited by 2,003
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806
A SWOT analysis of ChatGPT: Implications for educational practice and research
Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2195846 · Cited by 1,384
Cited by
Still being checked.
Metadata
- Authors
- Moen, M. N. M. B. A., Rohmatin, A., Nurazizah, M. S., & Kaleem, M.
- Affiliations
- International Islamic University Malaysia (MY)
- Venue
- International Journal of Applied Research and Innovation
- Publisher
- CV. Scripta Intelektual Mandiri
- Published
- 2026-10-20
- Volume
- 1
- Issue
- 2
- Pages
- 159-169
- ISSN
- 3124-3665 (ISSN-L)
- 3124-3630
- OpenAlex
- W7223092347
- Cited by
- 0
- References
- 26
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Journal of Applied Research and Innovation (publishedVersion, cc-by-sa)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- generative AI, higher education, learning outcomes, instructional design, cooperative learning, academic integrity, assessment redesign, learner metacognition, student motivation, self-efficacy, feedback, creative outcomes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


