Effects of Generative AI on Engagement and Performance Within Asynchronous Online Courses

AI snapshot

**TL;DR** — This mixed-methods, quasi-experimental study examined whether optional use of Spark, a Nectir AI course assistant embedded in every course at Los Angeles Pacific University, was associated with student engagement (cognitive, affective, behavioral) and course GPA in fully asynchronous online courses. The authors conclude that the key distinction is whether students used Spark at all rather than how intensively, and that Spark functions as a modest, complementary support rather than a primary driver of achievement or engagement. **Key findings** - **Cognitive:** Expected and actual GPA were strongly correlated for both groups — Spark users r(341) = .77, p < .001 (n = 343) and non-users r(1743) = .78, p < .001 (n = 1745) — so Spark use did not change grade-prediction accuracy. No statistically significant differences between users and non-users on perceived advisement, course organization, relevance, or clarity (Q9–Q12; all p ≥ .141), though non-users showed stronger inter-item correlations (r = .57–.79) than users (r = .50–.69). - **Performance:** Spark users had a significantly higher mean course GPA (M = 3.32, SD = 1.05, n = 341) than non-users (M = 3.04, SD = 1.23, n = 1,743), t = 4.39, p < .001, Cohen's d = 0.23 (mean difference = 0.28 GPA points). Usage frequency showed only weak linear relationships with GPA (message count r = .03; conversation count r = .01), and the Low/Medium/High usage tiers did not differ significantly, F(2,409) = 1.41, p = .24, η² = .01 (Low M = 3.15, n = 137; Medium M = 3.18, n = 138; High M = 3.36, n = 137). - **Affective:** Sentiment polarity was slightly but significantly more positive for Spark users (M = 0.038, SD = 0.13) than non-users (M = 0.019, SD = 0.12), t(432.6) = 2.88, p = .004 — a statistically significant but very small difference the authors describe as directional rather than a practical shift. - **Behavioral:** Spark usage correlated weakly and negatively with the three engagement items (Q1 r = –.06; Q3 r = –.07; Q5 r = –.05). Only Q3 differed significantly between groups, with Spark users reporting slightly *lower* scores, t(396.2) = –2.09, p = .037 (Q1 p = .076; Q5 p = .149). Longitudinally across seven terms (Spring 2024–Spring 2025), Spark users reported consistently higher engagement scores on all three items beginning in Fall 2024 and continuing into Spring 2025. **Methods** - Mixed-methods, quasi-experimental observational design without a randomized control group, comparing students who elected to use Spark with those who did not; all analyses are described as correlational. - Data from 1,335 unique students and 2,088 unique student-course combinations at Los Angeles Pacific University (a fully online, asynchronous, Hispanic-serving institution; ~42.6% Hispanic/Latino; female-to-male ratio ~4.32:1). Sources: end-of-course survey (Likert items plus open-ended responses), LMS-exported final grades converted to a 4.0 GPA scale, and Spark usage metrics (message/conversation counts and self-reported use, Q46). - Analyses: Pearson product-moment correlations (expected vs. actual GPA; survey item intercorrelations), independent-samples t-tests (survey items, GPA, sentiment), one-way ANOVA across usage tertiles, TextBlob lexicon-based sentiment polarity on concatenated open-ended responses, and longitudinal term-by-term comparison of engagement items. Internal consistency: Cronbach's α = 0.88 (Q9–Q12) and α = 0.94 (Q1, Q3, Q5). **Limitations** - Stated by the authors: Spark use was voluntary, so self-selection (e.g., higher motivation, self-regulation, or openness to technology) may confound results and causal claims are not warranted; data are limited to LAPU's internal records and to an asynchronous online environment; some courses require Spark for specific activities, and instructor promotion may affect willingness to use it; engagement is self-reported, and Q3 averaged 4.55/5, indicating a possible ceiling effect. - Also evident in the text: the design lacks a randomized control group and is explicitly correlational; usage was measured two different ways (self-reported Q46 for longitudinal analyses vs. system-logged counts for the GPA tier analysis), which the authors note may produce discrepancies; TextBlob's lexicon-based approach is acknowledged as insensitive to context, sarcasm, and domain-specific language; and the tier analysis is described as sample-specific, data-driven, and exploratory.