CoChat
Sign Up
Cited by 31Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningStudent Assessment and Feedback

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 13 of the 43 sources its claims cite were checked for support.

Socratic Mind: Impact of a Novel GenAI-Powered Formative Assessment Tool on Student Learning and Higher-Order Thinking

Open access
Lee, Jeonghyun, Hung, Jui-Tse, Yilmaz Soylu, Meryem, Popescu, Diana, Cui, Christopher Zhang, Grigoryan, Gayane, Joyner, David A., Harmon, Stephen W.·2026·Technology, Knowledge and Learning
DOI 10.1007/s10758-026-10007-6
Open this paper in CoChat — free accountOpen PDF

Abstract

This study examines the impact of a Generative Artificial Intelligence (GenAI)-powered formative assessment tool that employs Socratic questioning to support student learning in a large, fully online undergraduate-level computing course. Employing a quasi-experimental, mixed-methods design, we investigated participants’ engagement patterns, the influence of user experience on engagement, and impacts on both perceived and actual learning outcomes. Data were collected from the system logs, surveys on user experience and perceived engagement and learning gains, student reflections, and course performance data. Results indicated that participants consistently reported high levels of affective, behavioral, and cognitive engagement and these were strongly linked to positive user experiences and perceived learning outcomes. Quantitative analysis further revealed that students who engaged with the GenAI tool experienced significant gains in their quiz scores compared to those who did not, particularly benefiting students with lower baseline achievement. Additionally, thematic analysis of qualitative feedback revealed substantial perceived improvements in higher-order thinking skills, including problem-solving, critical thinking, and self-reflection. Our findings highlight the promise of AI-mediated dialogue in fostering deeper engagement and higher-order cognitive skills. As higher education institutions expand GenAI integration in curriculum, this dialogic, GenAI-powered assessment tool can offer a scalable strategy to promote students’ meaningful learning outcomes.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This quasi-experimental mixed-methods study examined a GenAI-powered Socratic formative assessment tool in a large online introductory computing course. It found high self-reported engagement and positive UX, a modest buffering effect on quiz-score decline for tool users—especially lower-performing students—and perceived gains in higher-order thinking, though perceived and actual learning were not consistently aligned.

Key findings

  • Students reported high engagement across dimensions: behavioral engagement M = 4.07 (SD = 0.61), affective M = 3.96 (SD = 0.76), and cognitive M = 3.96 (SD = 0.68); UX ratings were also positive for effectiveness M = 4.09 (SD = 1.08), usability M = 3.99 (SD = 1.16), and satisfaction M = 3.88 (SD = 1.23). UX subscales correlated positively with all engagement subscales, r = 0.33–0.54, p < 0.01.
  • In the difference-in-differences analysis, quiz performance declined overall by about 11.84 points from pre- to post-intervention, but engagement with the AI assignment reduced that decline by an estimated 3.30 points (β = 0.107, p < 0.05; Cohen’s f² = 0.31). No buffering effect was found for unit test scores (β = -0.053, p > 0.05).
  • Lower baseline achievement benefited more: the treatment-by-pre-score interaction was significant (β = -0.635, p < 0.05), and for every 1-point lower baseline score, the benefit of the tool increased by approximately 0.21 points (B = -0.217).
  • Perceived learning was strongly correlated with affective (r = 0.76), behavioral (r = 0.66), and cognitive engagement (r = 0.58), and the regression model explained about 69% of variance in perceived learning (R² = 0.685). However, in the Unit 3 group, perceived learning was negatively correlated with Unit 3 quiz (r = -0.30, p < 0.01) and test scores (r = -0.35, p < 0.01). Thematic analysis found perceived positive changes in problem-solving (69%), critical thinking (62%), self-reflection (60%), and verbal communication (38%).

Methods

  • Quasi-experimental, mixed-methods design in a 16-week fully online asynchronous undergraduate course, CS1301 Introduction to Computing, at a U.S. technology-focused research university. Of 579 enrolled students, students were randomly assigned to complete optional AI-based assignments for either Unit 3 or Unit 4; the final analytic sample was 171 students (81 Unit 3, 90 Unit 4) who completed both the assignment and post-assignment survey on time.
  • Data included system logs, engagement and learning scales, a UX survey, open-ended reflections, and academic performance data (quizzes, tests, final exam). Quantitative analyses used descriptive statistics, correlations, multiple regression, and difference-in-differences models in SPSS 29; qualitative analysis used thematic coding of 141 open-ended responses.

Limitations

  • The paper states the study was conducted in a single undergraduate computing course at one institution, limiting generalizability to other disciplines, contexts, or student populations.
  • Participation was optional and tied to extra credit, and the final sample included only students who completed both the assignment and post-survey, so self-selection may have biased results toward more motivated or engaged students; the quasi-experimental design cannot fully rule out selection bias or unmeasured confounding.
  • The paper states there was no direct performance-based assessment of higher-order thinking skills, so those gains are perceived rather than objectively measured; the short intervention-to-outcome interval may not capture skill development over time. It also notes technical issues and content compatibility barriers for audio interaction, and that high reliability alphas may reflect item redundancy.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10758-026-10007-6

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

13 of 43 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction · p. 2

    Current AI-based systems often emphasize lower-level cognitive skills, such as factual recall and procedural tasks, rather than fostering higher-order thinking abilities such as analysis, synthesis, and critical reasoning.

    “However, current AI-based systems often emphasize lower-level cognitive skills, such as factual recall and procedural tasks, rather than fostering higher-order thinking abilities such as analysis, synthesis, and critical reasoning (Foung et al., 2024 ; Roll &amp; Wylie, 2016 ).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Foung et al., 2024 source
    • cites Roll & Wylie, 2016 source

      The Learning Goals section (page unavailable) says the vast majority of reviewed papers focused on domain-level learning, and Linguistic Analysis describes a focus on step-based, well-defined problems. That supports the broad contrast, but the paper does not equate domain-level learning with factual recall or say that AI systems generally neglect analysis, synthesis, and critical reasoning; it also describes a system that asks students to synthesize documents. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation
    § 2.2 Socratic Questioning in Learning and Assessment: Potential Application of GenAI · p. 4

    Emerging research has investigated the potential of integrating GenAI technologies to scale and automate Socratic pedagogical strategies.

    “In response, emerging research has investigated the potential of integrating GenAI technologies as a robust opportunity to effectively scale and automate Socratic pedagogical strategies (Hung et al., 2024 ; Fakour &amp; Imani, 2025 ; Favero et al., 2024 ).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Hung et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Fakour & Imani, 2025 source
    • cites Favero et al., 2024 source
  • Premisecitation
    § 1 Introduction · p. 2

    Design features and User Experiences (UX) significantly influence the depth and quality of learner interaction.

    “Design features and User Experiences (UX) significantly influence the depth and quality of learner interaction (Chi &amp; Wylie, 2014; D'Mello &amp; Graesser, 2023 ; Sajja et al., 2025 ).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Chi & Wylie, 2014 source
    • cites D'Mello & Graesser, 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Sajja et al., 2025 source

      Section 3.6 (page unavailable) reports significant associations between students’ comfort with the AI tool, perceived homework usefulness, and how frequently they used it. The Discussions section also recommends task-aligned features and scaffolding. These passages support a link between user experience, design, and engagement, but they do not establish a significant effect on the depth or quality of learner interaction. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Interpretationcitation
    § 5.2 Impact of the AI Learning Tool on Students' Perceived and Actual Learning Outcomes · p. 20

    The scaffolded, conversational nature of the AI tool may have created a strong subjective sense of mastery that did not fully translate to improved performance under traditional assessment conditions.

    “the scaffolded, conversational nature of the AI tool may have created a strong subjective sense of mastery that did not fully translate to improved performance under traditional assessment conditions”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Carpenter et al., 2020 source
    • cites Deslauriers et al., 2019 source

      The Results and Discussion (page not supplied) say that fluent lectures can inflate students’ perceptions of learning even when independent test scores are lower than after active instruction. The paper does not study an AI tool or test whether conversational scaffolding creates that effect under traditional assessment conditions. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    Affective engagement was significantly higher in Unit 4 (M = 4.07, SD = 0.67) than in Unit 3 (M = 3.83, SD = 0.84), t(168) = -2.07, p = 0.04, Cohen's d = -0.32.

    “Affective | 3.96 (0.76) | 3.83 (0.84) | 4.07 (0.67) | 168 | - 2.07 | 0.04 | - 0.32”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim. 35
  • It may state two separate claims as one. 49

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Premisecitation
    § 2.3 Student Engagement and Learning with GenAI-Assisted Educational Tools · p. 5

    Many ChatGPT-focused studies suffer from methodological limitations, including unclear treatment descriptions, poorly matched control groups, and outcome measures that fail to capture durable learning.

    “Weidlich et al., ( 2025 ) argue that many ChatGPT-focused studies suffer from methodological limitations, including unclear treatment descriptions, poorly matched control groups, and outcome measures that fail to capture durable learning, which makes effect sizes difficult to interpret.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Weidlich et al., 2025 source

  • It may state two separate claims as one.

    Premisecitation
    § 1 Introduction · p. 2

    Socratic questioning enhances students' ability to question assumptions, articulate reasoning, and reflect on their understanding.

    “Research has demonstrated that engaging with Socratic dialogue enhances students' ability to question assumptions, articulate reasoning, and reflect on their understanding (Chin &amp; Osborne, 2008 ).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chin & Osborne, 2008 source

      The sections on classroom discourse and the quality of students’ questions link questioning to examining assumptions, explaining reasoning, and reflecting on ideas. The paper discusses student-generated questions and cooperative discussion, however, rather than demonstrating that Socratic dialogue specifically improves all three abilities. No page numbers are available in our copy. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 2.3 Student Engagement and Learning with GenAI-Assisted Educational Tools · p. 4

    Prior research suggests that User Experience (UX) with such tools, including ease of use, sense of usefulness, and responsiveness, substantially influence engagement dimensions.

    “Prior research suggests that User Experience (UX) with such tools, including ease of use, sense of usefulness, and responsiveness, substantially influence these engagement dimensions by shaping how students interact with educational content and perceive their value (Sajja et al., 2025 ).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Sajja et al., 2025 source

      Section 3.6 reports that comfort and perceived helpfulness were significantly associated with more frequent use of the AI assistant. It does not establish that UX causes engagement, and its convenience–usage correlation was not statistically significant. The cited findings do not show that responsiveness influences engagement. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 1 Introduction · p. 2

    Students' varying levels of AI literacy, prior experience, and confidence impact how they engage with AI tools.

    “Students' varying levels of AI literacy, prior experience, and confidence further impact how they engage with these tools (Guettala et al., 2024 ).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Guettala et al., 2024 source

      The Introduction discusses tailoring learning to students’ preferences and proficiencies, and the ‘Enhanced student engagement’ section links personalized content to engagement. Neither addresses whether students’ AI literacy, prior experience, or confidence affects how they engage with AI tools; searches for those terms found no supporting passage in our copy. (LLM, conf=0.74)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Premisecitation
    § 2.1 Integration of GenAI-Assisted Assessment Tools in Higher Education · p. 3

    Formative assessment is a continuous, process-oriented approach in which evidence about student learning is elicited, interpreted, and used to inform ongoing instructional decisions.

    “Formative assessment plays a central role in this shift, as it refers to a continuous, process-oriented approach in which evidence about student learning is elicited, interpreted, and used to inform ongoing instructional decisions (Black &amp; Wiliam, 2009 ).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Black & Wiliam, 2009 source

  • It may state two separate claims as one.

    Premisecitation
    § 2.1 Integration of GenAI-Assisted Assessment Tools in Higher Education · p. 2

    GenAI-empowered assessment systems dynamically adapt to individual learners by diagnosing misconceptions and suggesting personalized learning pathways.

    “GenAI-empowered assessment systems dynamically adapt to individual learners by utilizing sophisticated algorithms, diagnosing misconceptions, and suggesting personalized learning pathways (Heil et al., 2025 ; Zhu &amp; Zhang, 2025 ).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Heil et al., 2025 source

      The Introduction (page not provided) describes AI’s potential to personalise learning and provide adaptive support, while “Generative artificial intelligence for assessment” says GenAI-supported assessment may involve adaptive learning. Neither passage says these systems diagnose misconceptions or suggest personalised learning pathways; the paper presents adaptation as a possibility, not an established system capability. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Zhu & Zhang, 2025 source

  • It may state two separate claims as one.

    Premisecitation
    § 2.3 Student Engagement and Learning with GenAI-Assisted Educational Tools · p. 5

    Long-term reliance on LLMs is associated with reduced neural connectivity, originality, and perceived ownership of work.

    “For instance, Kosmyna et al., ( 2025 ) EEG study found that long-term reliance on LLMs was associated with reduced neural connectivity, originality, and perceived ownership of work, signaling potential erosion of deeper learning skills.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kosmyna et al., 2025 source

      The Abstract reports weaker neural connectivity and lower perceived ownership in the LLM group; the Conclusions also describe its essays as more homogeneous. The Discussion (p. 150) reports a narrower range of ideas after repeated LLM use, but calls that finding preliminary. The paper says longitudinal studies are needed to establish long-term effects, so it does not support the claim about long-term reliance as stated. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim. It may state two separate claims as one.

    Premisecitation
    § 1 Introduction · p. 2

    Existing studies report mixed findings on GenAI's impact on learning outcomes, requiring further empirical research.

    “So far though, existing studies report mixed findings on GenAI's impact on learning outcomes, which requires further empirical research (Bauer et al., 2025 ).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Bauer et al., 2025 source

      In “Recent Publication Trends in AI-Enhanced Learning” (page unavailable), the paper says GenAI-focused syntheses have few available studies and calls for continued research that accounts for methodological limitations. It cautions against broad conclusions about learning outcomes, but the passages do not establish that existing GenAI studies report mixed findings on those outcomes. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • It may state two separate claims as one.

    Findingown data
    § 4.2.1 Perceived Learning Outcomes · p. 13

    PL showed moderate significant correlations with usability (r=0.45, p<0.01), effectiveness (r=0.50, p<0.01), and satisfaction (r=0.54, p<0.01).

    “PL showed moderate, significant correlations with all three UX sub-scales: usability ( r = 0.45, p &lt; 0.01), effectiveness ( r = 0.50, p &lt; 0.01), and satisfaction ( r = 0.54, p &lt; 0.01).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.2 Actual Academic Performance Outcomes · p. 15

    Students overall experienced a decrease of approximately 11.84 points from pre- to post-intervention, but those who participated in the AI-based assignment saw this drop reduced by an estimated 3.30 points (Cohen's f2=0.31).

    “While students overall experienced a decrease of approximately 11.84 points on average from pre- to post-intervention, those who participated in the AI-based assignment saw this drop reduced by an estimated 3.30 points with a large overall effect size (Cohen's f 2 = 0.31)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    The median total completion time was 4,321 s (approximately 72 min).

    “The median 2 total completion time was 4,321 s (approximately 72 min).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    The mean total number of user message characters was 13,340.6 (roughly 2,668 words), with an average of 185.4 characters per message (about 37 words).

    “The mean total number of user message characters was 13,340.6 (roughly 2,668 words), with an average of 185.4 characters per message (about 37 words).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    The Unit 3 group generated a greater total number of user message characters (M = 10,993.4) than the Unit 4 group (M = 10,082.4), t(169) = 4.93, p < 0.001.

    “At the group level, the Unit 3 group generated a greater total number of user message characters (M = 10,993.4) than the Unit 4 group (M = 10,082.4), t (169) = 4.93, p &lt; 0.001.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.1 Perceived Learning Outcomes · p. 13

    Perceived learning (PL) was strongly positively correlated with AE (r=0.76, p<0.01), BE (r=0.66, p<0.01), and CE (r=0.58, p<0.01).

    “PL was strongly and positively correlated with all three engagement sub-scales: AE ( r = 0.76, p &lt; 0.01), BE ( r = 0.66, p &lt; 0.01), and CE ( r = 0.58, p &lt; 0.01).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.1 Perceived Learning Outcomes · p. 14

    AE (β=0.445, p<0.001) and BE (β=0.181, p<0.05) were the strongest predictors of PL, even after accounting for demographic factors, UX variables, and interaction behavior patterns.

    “Among all predictors, engagement subscales emerged as the strongest contributors, particularly AE ( β = 0.445, p &lt; 0.001) and BE ( β = 0.181, p &lt; 0.05), even after accounting for demographic factors, UX variables, and interaction behavior patterns.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 13

    User audio input count was positively correlated with CE (r=0.20, p<0.01), AE (r=0.26, p<0.01), and BE (r=0.22, p<0.01).

    “the user audio input count variable was also significantly and positively correlated with all three engagement sub-scales: CE ( r = 0.20, p &lt; 0.01), AE ( r = 0.26, p &lt; 0.01), and BE ( r = 0.22, p &lt; 0.01).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    The average number of audio input messages per question was comparable between the two groups (Unit 3: M = 2.2; Unit 4: M = 2.8).

    “Similarly, the average number of audio input messages per question was comparable between the two groups (Unit 3: M = 2.2; Unit 4: M = 2.8).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    On average, participants produced 83.6 messages across the entire assignment, with counts ranging from 32 to 237.

    “On average, participants produced 83.6 messages across the entire assignment, with counts ranging from 32 to 237.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.1 Perceived Learning Outcomes · p. 13

    Within the Unit 3 group, PL scores had significant negative correlations with both Unit 3 quiz (r=-0.30, p<0.01) and test scores (r=-0.35, p<0.01).

    “within the Unit 3 group we observed significant negative correlations between PL scores and both the Unit 3 quiz ( r = - 0.30, p &lt; 0.01) and test scores ( r = - 0.35, p &lt; 0.01).”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.1 Perceived Learning Outcomes · p. 13

    Three participation indicators had small positive correlations with PL: user audio input message count (r=0.16, p<0.05), total user message character count (r=0.17, p<0.05), and average user message character count (r=0.21, p<0.05).

    “three indicators of participation had small but significant, positive correlations with PL: user audio input message count ( r = 0.16, p &lt; 0.05), total user message character count ( r = 0.17, p &lt; 0.05), and average user message character count ( r = 0.21, p &lt; 0.05).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    In terms of UX, participants rated effectiveness most positively (M = 4.09, SD = 1.08), followed by usability (M = 3.99, SD = 1.16) and overall satisfaction (M = 3.88, SD = 1.23).

    “In terms of UX, the participants rated the effectiveness of the platform most positively (M = 4.09, SD = 1.08), followed by usability (M = 3.99, SD = 1.16) and overall satisfaction (M = 3.88, SD = 1.23).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    Approximately 30% of all messages were submitted via audio input.

    “Approximately 30% of all messages were submitted via audio input.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    Participants generally reported high levels of engagement and positive UX with the AI platform.

    “Participants generally reported high levels of engagement and positive UX with the AI platform (see Table 2 ).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    No statistically significant difference emerged in average message length between Unit 3 (M = 196.6) and Unit 4 (M = 175.3).

    “However, no statistically significant difference emerged in average message length between Unit 3 (M = 196.6) and Unit 4 (M = 175.3).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    Unit 3 participants produced more messages per question (M = 8.9) than Unit 4 participants (M = 7.9), t(169) = 2.90, p < 0.01.

    “When comparing message-level activity (per question), Unit 3 participants still produced more messages (M = 8.9) than Unit 4 participants (M = 7.9), t (169) = 2.90, p &lt; 0.01.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 11

    Behavioral engagement was the highest among the three engagement dimensions (M = 4.07, SD = 0.61), followed by affective engagement (M = 3.96, SD = 0.76) and cognitive engagement (M = 3.96, SD = 0.68).

    “Behavioral engagement (BE) was the highest among the three engagement dimensions (M = 4.07, SD = 0.61), followed closely by affective engagement (AE) (M = 3.96, SD = 0.76) and cognitive engagement (CE) (M = 3.96, SD = 0.68), indicating that students were consistently active and emotionally involved while using the tool.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.1 Patterns of Student Engagement and UX · p. 12

    UX sub-scale scores were generally consistent across assignment units, except for affective engagement (AE), where Unit 4 students reported significantly higher scores (M=4.07, SD=0.67) than Unit 3 students (M=3.83, SD=0.84), t(168)=-2.07, p=0.04.

    “perceptions of the tool were generally consistent regardless of the assignment unit. The only exception was AE, where students in Unit 4 reported significantly higher scores (M = 4.07, SD = 0.67) compared to those in Unit 3 (M = 3.83, SD = 0.84), t (168) = - 2.07, p = 0.04.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.2 Actual Academic Performance Outcomes · p. 15

    The DiD regression model was statistically significant for quiz (F(4,793)=86.02, p<0.001) and test (F(4,793)=61.11, p<0.001), accounting for approximately 30% and 24% of variance, respectively.

    “The overall model was statistically significant with F (4, 793) = 86.02, p &lt; 0.001 for the quiz model and F (4, 793) = 61.11, p &lt; 0.001 for test model, accounting for approximately 30% of the variance in quiz scores and 24% in test scores”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.1 Perceived Learning Outcomes · p. 14

    The multiple regression model predicting PL was statistically significant, F(18,142)=17.19, p<0.001, and explained approximately 69% of the variance in PL scores.

    “The overall model was statistically significant, F (18, 142) = 17.19, p &lt; 0.001, and explained approximately 69% of the variance in PL scores.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.3 The Influence of Socratic Questioning on Higher-Order Thinking Skills · p. 18

    5% of respondents described mixed experiences in verbal communication.

    “Additionally, 5% described mixed experiences”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.3 The Influence of Socratic Questioning on Higher-Order Thinking Skills · p. 17

    Students generally reported the strongest perceived gains in problem-solving, followed by critical thinking and self-reflection, with relatively limited changes in verbal communication.

    “Students generally reported the strongest gains in problem-solving, followed by notable perceived improvements in critical thinking and self-reflection, and relatively limited changes in verbal communication”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.2 Actual Academic Performance Outcomes · p. 16

    There was a significant negative interaction between pre-intervention scores and treatment (β=-0.635, p<0.05); for every 1-point lower baseline score, the benefit of the assessment tool increased by approximately 0.21 points (B=-0.217).

    “Notably, there was a significant negative interaction between pre-intervention scores and treatment (β = - 0.635, p &lt; 0.05). Based on the unstandardized coefficient (B = - 0.217), for every 1-point lower baseline score, the benefit of the assessment tool increased by approximately 0.21 points.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 5.3 Potential Influence of the AI Learning Tool on Students' Higher-Order Thinking Skills · p. 21

    Qualitative feedback suggested that students found the structured, Socratic-style interactions beneficial for moving beyond surface-level understanding, developing analytical thinking, and clarifying logic in programming tasks.

    “Qualitative feedback further suggested that students found the structured, Socratic-style interactions as beneficial for moving beyond surface-level understanding, developing analytical thinking and clarifying logic in programming tasks.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.2.2 Actual Academic Performance Outcomes · p. 16

    The extended DiD model was statistically significant, F(5,792)=69.99, p<0.001, accounting for 31% of variance in quiz scores (R2=0.306), with a large effect size (Cohen's f2=0.44).

    “The overall model was statistically significant, F (5, 792) = 69.99, p &lt; 0.001, and accounted for 31% of the variance in quiz scores ( R² = 0.306), which corresponds to a large effect size (Cohen's f 2 = 0.44)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given
    § 6 Limitations · p. 22

    Oral interaction appeared to enhance cognitive engagement.

    “Finally, while oral interaction appeared to enhance cognitive engagement, several barriers emerged.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given
    § Appendix A. Illustration of Student Interaction with the GenAIpowered Assessment Tool · p. 23

    The interaction lasted a total of 4 min and 13 s.

    “The interaction lasted a total of 4 min and 13 s.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 7 Conclusion · p. 22

    Students reported high levels of affective, behavioral, and cognitive engagement, and generally perceived the tool as usable and beneficial.

    “Students reported high levels of affective, behavioral, and cognitive engagement, and generally perceived the tool as usable and beneficial.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Self-reported engagement and perceived learning reflect actual engagement and learning.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.3 The Influence of Socratic Questioning on Higher-Order Thinking Skills · p. 18

    About 1% of respondents provided mixed feedback on self-reflection.

    “About 1% provided mixed feedback”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 4.3 The Influence of Socratic Questioning on Higher-Order Thinking Skills · p. 18

    56% of participants indicated no perceived change in verbal communication, largely due to typing responses rather than using the AI's verbal interaction feature.

    “more than half of the participants (56%) indicated no perceived change in verbal communication, largely due to their decision to type responses rather than utilize the AI's verbal interaction feature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.3 The Influence of Socratic Questioning on Higher-Order Thinking Skills · p. 18

    5% of respondents expressed mixed experiences in critical thinking.

    “Additionally, 5% expressed mixed experiences”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 7 Conclusion · p. 23

    Students who engaged with the tool experienced a significantly smaller decline in quiz scores over time, especially those with lower prior achievement.

    “Notably, students who engaged with the tool experienced a significantly smaller decline in quiz scores over time, especially those with lower prior achievement”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Quiz score decline is a valid measure of learning.

  • It may state two separate claims as one.

    Findingnone given
    § 6 Limitations · p. 22

    Some students experienced technical issues or preferred text-based formats due to comfort or environmental constraints.

    “Some students experienced technical issues or preferred text-based formats due to comfort or environmental constraints.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationown data
    § 4.1 Patterns of Student Engagement and UX · p. 12

    The higher AE in Unit 4 suggests students may have experienced greater emotional connection or interest during the Unit 4 assessment experience.

    “This finding suggests that students may have experienced a greater sense of emotional connection or interest during the Unit 4 assessment experience.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 2.3 Student Engagement and Learning with GenAI-Assisted Educational Tools · p. 4

    Student engagement is a critical factor determining the effectiveness of AI-driven educational tools.

    “Student engagement is a critical factor determining the effectiveness of AI-driven educational tools.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Chi & Wylie, 2014 source
    • cites D'Mello & Graesser, 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given
    § 3.3.4 Academic Performance Scores · p. 10

    All quizzes and tests were timed and proctored using Honorlock.

    “All quizzes and tests were timed and proctored using Honorlock.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Study design and conductreasoning
    § 3.2 Procedures · p. 7

    All students were assigned to use the AI tool at some point during the course, so the comparison reflects earlier versus later exposure to the GenAI-powered tool.

    “In other words, all students were assigned to use the AI tool at some point during the course; thus, the comparison reflects earlier versus later exposure to the GenAIpowered tool.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Premisecitation
    § 2.2 Socratic Questioning in Learning and Assessment: Potential Application of GenAI · p. 4

    Regular exposure to Socratic dialogue enhances students' analytical and argumentative capacities and promotes deeper exploration of content.

    “Regular exposure to Socratic dialogue enhances students' analytical and argumentative capacities and promotes deeper exploration of content (Paul &amp; Elder, 2006 ; Yang et al., 2005 ).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Paul & Elder, 2006

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Yang et al., 2005 source

  • Probably not a substantial claim.

    Premisecitation
    § 1 Introduction · p. 2

    GenAI platforms integrating Socratic questioning hold promise for addressing scalability by delivering scalable and adaptive assessments.

    “GenAI platforms integrating Socratic questioning hold promise for addressing this challenge by delivering scalable and adaptive assessments (Authors, 2024 ).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Authors, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 1 Introduction · p. 2

    Learner engagement-cognitive, emotional, and behavioral-plays a critical role in the successful use of AI tools.

    “Learner engagement-cognitive, emotional, and behavioral-plays a critical role in the successful use of AI tools (Heung &amp; Chiu, 2025 ).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Heung & Chiu, 2025 source

  • It may state two separate claims as one.

    Study design and conductown data
    § 3.3.1 Engagement and Learning Scale · p. 8

    The engagement and learning subscales showed internal reliability: α = 0.90 for affective engagement, α = 0.76 for behavioral engagement, α = 0.85 for cognitive engagement, and α = 0.86 for perceived learning.

    “In addition, the engagement and learning sub-scales showed robust internal reliability scores, indicated by Cronbach's Alpha, with α = 0.90 for affective engagement; α = 0.76 for behavioral engagement; α = 0.85 for cognitive engagement; and α = 0.86 for perceived learning.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § Abstract · p. 1

    The findings highlight the promise of AI-mediated dialogue in fostering deeper engagement and higher-order cognitive skills.

    “Our findings highlight the promise of AI-mediated dialogue in fostering deeper engagement and higher-order cognitive skills.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 2.1 Integration of GenAI-Assisted Assessment Tools in Higher Education · p. 2

    GenAI is capable of producing human-like feedback that closely resembles human responses by leveraging large language models.

    “GenAI is capable of producing human-like feedback that closely resembles human responses by leveraging the capabilities of large language models (LLMs) (Foung et al., 2024 ; Weng et al., 2024 ).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Foung et al., 2024 source
    • cites Weng et al., 2024 source

  • Probably not a substantial claim.

    Premisecitation
    § 2.2 Socratic Questioning in Learning and Assessment: Potential Application of GenAI · p. 3

    Higher-order thinking skills encompass complex processes such as analysis, evaluation, and creation.

    “Higher-order thinking skills encompass complex processes such as analysis, evaluation, and creation (Anderson &amp; Krathwohl, 2001 ).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Anderson & Krathwohl, 2001

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • It may state two separate claims as one.

    Study design and conductown data
    § 3.3.2 UX Survey · p. 10

    The UX survey subscales showed internal consistency: α = 0.95 for usability and α = 0.97 for perceived learning effectiveness.

    “Internal consistency was strong across all three subscales (α = 0.95. for usability and α = 0.97. for perceived learning effectiveness).”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Premisecitation
    § 2.3 Student Engagement and Learning with GenAI-Assisted Educational Tools · p. 4

    Personalization enhances engagement, as AI tools tailor content and feedback to individual learner needs, thereby increasing motivation, self-efficacy, and perceived relevance.

    “Personalization further enhances engagement, as AI tools tailor content and feedback to individual learner needs, thereby increasing motivation, self-efficacy, and perceived relevance (Kong &amp; Yang, 2024 ; Yilmaz &amp; Yilmaz, 2023 ).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Kong & Yang, 2024 source
    • cites Yilmaz & Yilmaz, 2023 source

  • It may state two separate claims as one.

    Interpretationown data
    § 4.2.2 Actual Academic Performance Outcomes · p. 15

    The reduction represents approximately 25-30% of the observed performance decline, suggesting a meaningful reduction in performance loss rather than an absolute gain.

    “This represents approximately 25-30% of the observed performance decline, suggesting a meaningful reduction in performance loss rather than an absolute gain.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 4.2.2 Actual Academic Performance Outcomes · p. 15

    The decline in quiz performance is consistent with the increasing difficulty of the course content in Units 3 and 4.

    “which is consistent with the increasing difficulty of the course content in Units 3 and 4.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The course content became more difficult in later units (Units 3 and 4).

  • Probably not a substantial claim. It may state two separate claims as one.

    Study design and conductnone given
    § 3.2 Procedures · p. 7

    The two assignments were designed to be parallel in structure and assessment format, each aligned with the corresponding unit quiz topics.

    “Each assignment was aligned with the corresponding unit quiz topics, with both assignments designed to be parallel in structure and assessment format.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning
    § Abstract · p. 1

    As higher education institutions expand GenAI integration in curriculum, this dialogic, GenAI-powered assessment tool can offer a scalable strategy to promote students' meaningful learning outcomes.

    “As higher education institutions expand GenAI integration in curriculum, this dialogic, GenAI-powered assessment tool can offer a scalable strategy to promote students' meaningful learning outcomes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductreasoning
    § 6 Limitations · p. 21

    Motivation may have interacted with engagement patterns observed in this study, potentially amplifying the relationship between tool use and learning outcomes, though these effects are difficult to disentangle in the current study design.

    “In addition, motivation may have interacted with engagement patterns observed in this study, potentially amplifying the relationship between tool use and learning outcomes, though these effects are difficult to disentangle in the current study design.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationown data
    § Appendix A. Illustration of Student Interaction with the GenAIpowered Assessment Tool · p. 23

    The exchange illustrates a dynamic, multi-turn Socratic dialogue in which AI guides the student to refine their understanding of error handling in Python, particularly the use of try/except blocks.

    “This exchange illustrates a dynamic, multi-turn Socratic dialogue in which AI guides the student to refine their understanding of error handling in Python, particularly the use of try/except blocks.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § 5.2 Impact of the AI Learning Tool on Students' Perceived and Actual Learning Outcomes · p. 20

    The buffering effect was particularly noticeable among lower-performing students, suggesting that GenAI-supported Socratic dialogue can help close performance gaps in challenging content areas.

    “This buffering effect was particularly noticeable among lower-performing students, suggesting that GenAI-supported Socratic dialogue can help close performance gaps in challenging content areas.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Study design and conductown data
    § 5.3 Potential Influence of the AI Learning Tool on Students' Higher-Order Thinking Skills · p. 21

    These findings should be interpreted with caution because they rely on self-reported perceptions rather than direct performance-based measures of higher-order thinking.

    “However, these findings should be interpreted with caution, as they rely on self-reported perceptions rather than direct performance-based measures of higher-order thinking.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given
    § Declarations · p. 28

    Informed consent was obtained from all participants involved in the study.

    “Informed consent was obtained from all participants involved in the study.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Premisecitation
    § 5.2 Impact of the AI Learning Tool on Students' Perceived and Actual Learning Outcomes · p. 20

    Interactive and guided dialogue can reduce extraneous load during learning but may also reduce students' perceived effort without necessarily promoting the deeper processing required for durable knowledge construction.

    “while interactive and guided dialogue can reduce extraneous load during learning, it may also reduce students' perceived effort without necessarily promoting the deeper processing required for durable knowledge construction”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Sweller, 2020

  • It may state two separate claims as one.

    Interpretationcitation
    § 7 Conclusion · p. 23

    Compared to many commercially available GenAI platforms, this tool grants instructors meaningful pedagogical agency by allowing them to upload lecture-specific materials, design rubric-based questions and guided answers, and intentionally shape students' learning trajectories.

    “Importantly, compared to many commercially available GenAI platforms, this tool grants instructors meaningful pedagogical agency by allowing them to upload lecture-specific materials, design rubric-based questions and guided answers, and intentionally shape students' learning trajectories.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Study design and conductnone given
    § 6 Limitations · p. 21

    Participation in the AI-based assignment was optional and tied to extra credit, and the final analytic sample included only students who completed both the assignment and the post-survey.

    “Also, participation in the AI-based assignment was optional and tied to extra credit, and the final analytic sample included only students who completed both the assignment and the post-survey.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § 7 Conclusion · p. 22

    The GenAI-powered assessment tool effectively promoted meaningful engagement and supported learning in the context of introductory computer science education in higher education.

    “Our study demonstrates that the GenAI-powered assessment tool effectively promoted meaningful engagement and supported learning in the context of introductory computer science education in higher education.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Quiz score decline is a valid measure of learning.
    • Rests on an assumption: Self-reported engagement and perceived learning reflect actual engagement and learning.
    • Rests on an assumption: The observed association between tool use and quiz score decline is causal.
    • Rests on an assumption: The sample is representative enough to generalize to introductory CS education in higher education.

  • It may state two separate claims as one.

    Interpretationcitation
    § 7 Conclusion · p. 23

    This educator-driven design ensures that GenAI-facilitated feedback is both contextually relevant and pedagogically meaningful while addressing concerns about generic or uncontextualized AI outputs.

    “This educator-driven design, grounded on the humanAI partnership, ensures that GenAI-facilitated feedback is both contextually relevant and pedagogically meaningful while addressing concerns about generic or uncontextualized AI outputs (Luo et al., 2024 ).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Luo et al., 2024 source

  • Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

    Interpretationreasoning
    § 7 Conclusion · p. 23

    Such tools can complement-not replace-traditional assessments by providing personalized, dialogic feedback that encourages reflection and reasoning.

    “For educators, the results suggest that such tools can complement-not replace-traditional assessments by providing personalized, dialogic feedback that encourages reflection and reasoning.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Self-reported engagement and perceived learning reflect actual engagement and learning.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductreasoning
    § 6 Limitations · p. 22

    The relatively short duration between the intervention and outcome assessments may not adequately capture the development of higher-order thinking skills, which typically evolve over extended periods of sustained practice.

    “Furthermore, the relatively short duration between the intervention and outcome assessments may not adequately capture the development of higher-order thinking skills, which typically evolve over extended periods of sustained practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductnone given
    § Declarations · p. 28

    This study was conducted in accordance with ethical standards and received approval from the Georgia Institute of Technology Institutional Review Board.

    “This study was conducted in accordance with ethical standards and received approval from the Georgia Institute of Technology Institutional Review Board.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

    Interpretationreasoning
    § 7 Conclusion · p. 23

    By combining AI's adaptive capabilities with intentional instructional design, the tool fosters not only deeper cognitive engagement but also a stronger alignment between technology and pedagogy.

    “By combining AI's adaptive capabilities with intentional instructional design, the tool fosters not only deeper cognitive engagement but also a stronger alignment between technology and pedagogy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 7 Conclusion · p. 23

    The study extends work on dialogic learning by illustrating how Socratic questioning can be operationalized at scale through GenAI in a large, online course.

    “Beyond these empirical findings, our study extends work on dialogic learning by illustrating how Socratic questioning can be operationalized at scale through GenAI in a large, online course.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Study design and conductreasoning
    § 6 Limitations · p. 22

    No direct performance-based assessment of higher-order thinking skills was administered; therefore, the results reflect perceived rather than objectively measured cognitive gains.

    “It should be noted that no direct performance-based assessment of higher-order thinking skills was administered; therefore, these results should be interpreted as reflections of perceived rather than objectively measured cognitive gains.”
    Quote in the paperSubstantial claim
75 of 75

  • The paper relies on this without support.

    Traditional implementations of Socratic questioning require intensive instructor facilitation, limiting scalability.

    “Yet, traditional methods of implementing Socratic dialogue require extensive instructor involvement, limiting scalability.”

  • The paper relies on this without support.

    The AI tool's adaptive questioning accurately detects satisfactory conceptual clarity.

    “This AI tool works by prompting students with instructor-defined opening questions and adapting its follow-up inquiries based on the students' spoken or written responses until the system detects satisfactory conceptual clarity.”

  • The paper relies on this without support.

    The quizzes and tests validly measure learning outcomes.

    “As an indicator of learning outcomes, we collected participants' academic performance data, including their quiz scores from Units 2-5, test scores from Units 2-4, and their final exam scores.”

  • The paper relies on this without support.

    The survey items validly measure engagement and perceived learning.

    “This scale was designed to measure participants' perceived levels of engagement and learning outcomes after interacting with the GenAI tool.”

  • The paper relies on this without support.

    The thematic analysis of open-ended responses accurately captures perceived changes in higher-order thinking.

    “Qualitatively, thematic analysis of 141 open-ended responses was conducted to assess perceived changes in problem-solving, critical thinking, self-reflection, and verbal communication.”

  • The paper relies on this without support.

    The parallel trends assumption for difference-in-differences holds (i.e., in the absence of the intervention, the Unit 3 and Unit 4 groups would have followed parallel trends in test performance).

    “The DiD approach compared test performance over time between the treatment and comparison groups to estimate the causal effect of the AI intervention while controlling for baseline performance.”

  • The paper relies on this without support.

    The AI tool's feedback is accurate and helpful.

    “Upon completion, the platform delivers immediate, personalized feedback highlighting strengths and areas for improvement in the overall performance.”

  • The paper relies on this without support.

    In the absence of the AI intervention, the Unit 3 and Unit 4 groups would have followed parallel trends in quiz/test performance.

    “using a DiD approach, we conducted a multiple regression analysis to further examine the effect of engagement with the AI-driven assessment tool on participants' academic performance over time, specifically comparing scores before and after the assignment intervention.”

  • The paper relies on this without support.

    The course content became more difficult in later units (Units 3 and 4).

    “which is consistent with the increasing difficulty of the course content in Units 3 and 4.”

  • The paper relies on this without support.

    Quiz score decline is a valid measure of learning.

    “Notably, students who engaged with the tool experienced a significantly smaller decline in quiz scores over time, especially those with lower prior achievement”

  • The paper relies on this without support.

    Self-reported engagement and perceived learning reflect actual engagement and learning.

    “Students reported high levels of affective, behavioral, and cognitive engagement, and generally perceived the tool as usable and beneficial.”

  • The paper relies on this without support.

    The observed association between tool use and quiz score decline is causal.

    “suggesting a buffering effect that helped struggling learners.”

  • The paper relies on this without support.

    The sample is representative enough to generalize to introductory CS education in higher education.

    “Our study demonstrates that the GenAI-powered assessment tool effectively promoted meaningful engagement and supported learning in the context of introductory computer science education in higher education.”
13 of 13

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Final dataset is reported as both 173 and 171 students

    Major

    The methods give two different sizes for the final dataset. The latter figure is also the sum of the reported Unit 3 and Unit 4 group sizes.

  • Overall mean message-character count exceeds both group means

    Minor

    The reported overall mean of 13,340.6 total message characters cannot be the combined mean of groups whose corresponding means are 10,993.4 and 10,082.4.

  • English-language regression coefficient differs from Table 4

    Minor

    The narrative reports a standardized coefficient of 0.156 for native English language, whereas Table 4 reports 0.137 for English.

3 of 3

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc.
  2. Alavi, M., & Westerman, G. (2023). How generative AI will transform knowledge work. Harvard Business Review, 7.
  3. Collins, R. (2014). Skills for the 21st Century: Teaching higher-order thinking. Curriculum & Leadership Journal, 12(14), 1–8.
  4. Paul, R., & Elder, L. (2006). Critical thinking: The nature of critical and creative thought. Journal of Developmental Education, 30(2), 34. https://api.semanticscholar.org/CorpusID:151509960

Cited by

Sorted by citations

Related

Metadata

Authors
Lee, J., Hung, J.-T., Yilmaz Soylu, M., Popescu, D., Cui, C. Z., Grigoryan, G., Joyner, D. A., & Harmon, S. W.
Affiliations
  • Georgia Institute of Technology (US)
  • University of California San Diego (US)
Corresponding author
Lee, Jeonghyun
ORCID
Venue
Technology, Knowledge and Learning
Publisher
Springer Science and Business Media LLC
Published
2026-07-08
ISSN
  • 2211-1662 (ISSN-L)
  • 2211-1670
DOI
10.1007/s10758-026-10007-6
OpenAlex
W7167688959
Cited by
23
References
43
Citations by year
2026: 21
FWCI
131.56
Type
journal article
Language
en
Open access
Licence
Locations
  • Technology Knowledge and Learning (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Social Sciences › Education › Student Assessment and Feedback
Keywords
generative AI, formative assessment, Socratic questioning, student engagement, higher-order thinking skills, user experience, problem solving, critical thinking, self-reflection, learning gains, low-achieving students

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review