Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 2Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in Healthcare and Education

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 2 of the 30 sources its claims cite were checked for support.

Generative AI and Learning Dynamics in Higher Education: A Longitudinal Empirical Study

Open access
Melanou, Chrysanthi, Beege, Maik, Kimmig, Martin·2026·Journal of Computer Assisted Learning

Abstract

Background: Generative AI tools such as ChatGPT are increasingly integrated into higher education. Empirical evidence on how they shape learning dynamics over time remains limited. Previous studies have often examined single learning outcomes, neglecting the interplay between cognitive, motivational, and metacognitive processes that evolve throughout the semester. Objectives: This longitudinal study investigated how generative AI influences students' knowledge gain, motivation, cognitive load, critical thinking, and reflective use across an entire semester. It further explored whether prior knowledge and AI experience moderated these learning effects, addressing concerns about potential Matthew effects. Methods: A quasi‐experimental design was implemented with three parallel classes of business informatics students ( N = 87) assigned to different instructional conditions: tutor‐AI use, unguided AI, and a control group without AI. Data were collected at three measurement points (T1–T3). Results and Conclusion: Students showed significant knowledge gains across all groups, but no additional benefits of AI integration. Motivation remained largely stable; germane cognitive load increased and was positively associated with knowledge gain. Critical thinking remained stable, while reflective use was higher in the AI condition and positively associated with critical thinking. No evidence emerged for a Matthew effect, indicating that AI did not intensify performance inequalities. Overall, the findings did not support the view of generative AI as a consistent learning outcome. Its impact depended on pedagogical framing and learners' metacognitive regulation, underscoring the need for structured guidance and reflective engagement in AI‐supported education.

Overview

CoChat Summary

TL;DR

This longitudinal quasi-experiment with 87 business informatics students found significant knowledge gains across all conditions but no additional benefit of generative AI integration. AI’s impact depended on pedagogical framing and learners’ metacognitive regulation, with reflective use higher in AI conditions and germane cognitive load predicting knowledge gain.

Key findings

  • Knowledge increased from T1 to T3 across all groups, F(1, 50) = 29.87, p < 0.001, η²p = 0.374; there was no significant Group or Time × Group effect (p = 0.682 and p = 0.469), and no moderation by prior achievement or prior AI experience (BF01 = 8.70), indicating no Matthew effect.
  • Motivation remained stable (no Time effect, p = 0.467) and did not predict knowledge gain (R² = 0.010, p = 0.533). Germane cognitive load increased from T2 to T3, F(1, 37) = 10.86, p = 0.002, η²p = 0.227; GCL (β = 0.51, p = 0.001) and ECL (β = 0.36, p = 0.018) predicted knowledge gain.
  • Critical thinking remained stable and did not predict knowledge gain (p = 0.204) or cognitive load. Reflective use was higher in the AI-supported condition than control, F(1, 40) = 20.21, p < 0.001, η²p = 0.336, and positively associated with critical thinking (β = 0.43, p < 0.001), but did not predict knowledge gain (p = 0.652).
  • Of 16 hypotheses tested, six were supported and one partially supported.

Methods

  • Quasi-experimental longitudinal design across one semester with three parallel classes of second-semester Business Informatics students (N = 87; 19 females, 68 males; age 18–29), assigned to tutor-AI, unguided AI, or control conditions; the same instructor and materials were used, and students completed weekly individual in-class question-generation tasks over a nine-week theory phase.
  • Data were collected at T1 (first lecture), T2 (fourth lecture), and T3 (eighth lecture) via anonymised LimeSurvey questionnaires; knowledge was assessed with a 14-item multiple true-false test (maximum 70 points, converted to percentages), alongside self-report measures of motivation, cognitive load (ICL, ECL, GCL), critical thinking, and reflective use.
  • Analyses in JASP included repeated-measures ANOVA/ANCOVA, linear regressions, and PROCESS moderation/mediation models; groups were compared either as three conditions or as AI versus control depending on focus and power, with α = 0.05 and complete-case handling.

Limitations

  • The paper states that the quasi-experimental design assigned participants by pre-existing class groups rather than random allocation, so class-level confounds and causal caution remain; attrition was high and uneven (T3 missing relative to T1: Group C 38.1%, Group B 25.9%, Group A 12.1%), and complete-case analysis reduced statistical power.
  • The paper states further limitations: small and uneven group sizes limited detection of interaction effects; the single business informatics, practice-oriented programme limits generalisability; the approximately three-month semester may be too short for critical thinking or durable AI literacy; no explicit scaffolding or structured prompting was included; novelty/Hawthorne effects may have influenced engagement; and self-report measures were used, with the motivation scale showing low internal consistency (α ≈ 0.50).
  • Evident from the text: sample sizes vary widely by construct and measurement point (e.g., ECL T2 N = 33 vs. T3 N = 56; knowledge T1 N = 81 vs. T3 N = 62), and analyses use different complete-case Ns, which complicates cross-construct comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1002/jcal.70322

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

35 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Figures

FIGURE 1 CONSORT-­ style flow diagram illustrating participant enrolment, allocation to instructional conditions, valid cases across measurement points (T1, T2, T3), and attrition relative to T1.
FIGURE 1 CONSORT-­ style flow diagram illustrating participant enrolment, allocation to instructional conditions, valid cases across measurement points (T1, T2, T3), and attrition relative to T1.CC-BY · From the PDF
FIGURE 2 Development of knowledge scores from T1 to T3 and conditions (Group A: Tutor-­ AI, Group B: Control, Group C: Unguided AI).
FIGURE 2 Development of knowledge scores from T1 to T3 and conditions (Group A: Tutor-­ AI, Group B: Control, Group C: Unguided AI).CC-BY · From the PDF
FIGURE 3 Trajectory of motivation across the semester for three instructional conditions (Group A: Tutor-­ AI, Group B: Control, Group C: Unguided AI).
FIGURE 3 Trajectory of motivation across the semester for three instructional conditions (Group A: Tutor-­ AI, Group B: Control, Group C: Unguided AI).CC-BY · From the PDF
FIGURE 4 Development of cognitive load dimensions from T2 to T3 by condition ((0: Control, 1: AI): (a) Intrinsic cognitive load (ICL), (b) extraneous cognitive load (ECL), and (c) germane cognitive load (GCL)).
FIGURE 4 Development of cognitive load dimensions from T2 to T3 by condition ((0: Control, 1: AI): (a) Intrinsic cognitive load (ICL), (b) extraneous cognitive load (ECL), and (c) germane cognitive load (GCL)).CC-BY · From the PDF

Verification

Claims the paper makes

1 Valid · 1 Not valid · 23 Not checked · 163 Own results and interpretations

103 more claims

Create a free account to see all 188 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 56
  • Probably not a substantial claim. 38

Create a free account to see these claims.

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

2 of 2

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upF(1, 39) = 0.15reported p = .693 · recomputed p = .701

    “F(1, 39) = 0.15, p = .693”

  • Doesn’t add upF(1, 39) = 0.82reported p = .204 · recomputed p = .371

    “F(1, 39) = 0.82, p = .204”

  • Not confirmedF(1, 39) = 0.01reported p = .945 · recomputed p = .921

    “F(1, 39) = 0.01, p = .945”

  • ConsistentF(1, 50) = 29.87reported p < .001 · recomputed p < .001

    “F(1, 50) = 29.87,p < .001”

  • ConsistentF(2, 50) = 0.39reported p = .682 · recomputed p = .679

    “F(2, 50) = 0.39, p = .682”

5 of 35 · the rest aren’t loaded

30 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Research works printed but not linked: 5

  1. Hayes A. F., 2022, Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression‐Based Approach
  2. Liu Y. W.Cui S.Wu andW.Kek Tan.2025.“Generative AI in Education: A Review of Applications Impacts and Future Research Directions.”ICIS 2025 Proceedings (International Conference on Information Systems 2025).https://aisel.aisnet.org/icis2025/learn_curricula/learn_curricula/13.
  3. Melanou C., 2026, Transforming Education With AI
  4. Rigney D., 2010, The Matthew Effect: How Advantage Begets Further Advantage
  5. Wong A. H. L. W.Wong S. C.Hooi andS. S.Lee.2024.Student‐Generated Questions: A Novel Approach for Encouraging Cognitive EngagementHigher Education Conference in Singapore (HECS) 2024. National University of Singapore.https://blog.nus.edu.sg/hecs/hecs2024‐ahpwong‐et‐al‐2/.

Metadata

Authors
Melanou, C., Beege, M., & Kimmig, M.
Affiliations
  • Baden-Wuerttemberg Cooperative State University (DE)
  • University of Education Freiburg (DE)
Corresponding author
Melanou, Chrysanthi
Venue
Journal of Computer Assisted Learning
Publisher
Wiley
Published
2026-09-10
Volume
42
Issue
5
ISSN
  • 0266-4909 (ISSN-L)
  • 1365-2729
OpenAlex
W7212183669
Cited by
2
References
56
Citations by year
2026: 2
FWCI
9.92
Type
journal article
Language
en
Open access
Licence
Locations
Journal of Computer Assisted Learning (publishedVersion, cc-by)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
generative AI in education, ChatGPT, higher education, knowledge improvement, student motivation, cognitive load, critical thinking, Matthew effect, prior knowledge, AI experience, metacognitive regulation, business informatics education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.