CoChat
Sign Up
Cited by 463Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in Healthcare and Education

RETRACTED ARTICLE: The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis

Open access
Wang, Jin, Fan, Wenxiang·2025·Humanities and Social Sciences Communications
DOI 10.1057/s41599-025-04787-y
Open this paper in CoChat — free accountOpen PDF

Abstract

As a new type of artificial intelligence, ChatGPT is becoming widely used in learning. However, academic consensus regarding its efficacy remains elusive. This study aimed to assess the effectiveness of ChatGPT in improving students’ learning performance, learning perception, and higher-order thinking through a meta-analysis of 51 research studies published between November 2022 and February 2025. The results indicate that ChatGPT has a large positive impact on improving learning performance ( g = 0.867) and a moderately positive impact on enhancing learning perception ( g = 0.456) and fostering higher-order thinking ( g = 0.457). The impact of ChatGPT on learning performance was moderated by type of course ( Q B = 64.249, P < 0.001), learning model ( Q B = 76.220, P < 0.001), and duration ( Q B = 55.998, P < 0.001); its effect on learning perception was moderated by duration ( Q B = 19.839, P < 0.001); and its influence on the development of higher-order thinking was moderated by type of course ( Q B = 7.811, P < 0.05) and the role played by ChatGPT ( Q B = 4.872, P < 0.05). This study suggests that: (1) appropriate learning scaffolds or educational frameworks (e.g., Bloom’s taxonomy) should be provided when using ChatGPT to develop students’ higher-order thinking; (2) the broad use of ChatGPT at various grade levels and in different types of courses should be encouraged to support diverse learning needs; (3) ChatGPT should be actively integrated into different learning modes to enhance student learning, especially in problem-based learning; (4) continuous use of ChatGPT should be ensured to support student learning, with a recommended duration of 4–8 weeks for more stable effects; (5) ChatGPT should be flexibly integrated into teaching as an intelligent tutor, learning partner, and educational tool. Finally, due to the limited sample size for learning perception and higher-order thinking, and the moderately positive effect, future studies with expanded scope should further explore how to use ChatGPT more effectively to cultivate students’ learning perception and higher-order thinking.

Verified 2026-10-06 · 1 reference not checked yet

AI snapshot

Written from the paper’s full text.

TL;DR

This PRISMA-guided meta-analysis of 51 experimental and quasi-experimental studies (November 2022–February 2025) finds that ChatGPT has a large positive effect on students' learning performance (g = 0.867) and moderate positive effects on learning perception (g = 0.456) and higher-order thinking (g = 0.457). Effects are moderated by course type, learning model, duration, and the role ChatGPT plays, leading the authors to recommend scaffolded, sustained (4–8 week) integration of ChatGPT across grade levels and course types.

Key findings

  • Overall effects across 72 independent effect sizes: learning performance g = 0.867 (k = 44, 95% CI [0.683, 1.052], p < 0.001); learning perception g = 0.456 (k = 19, 95% CI [0.287, 0.625], p < 0.001); higher-order thinking g = 0.457 (k = 9, 95% CI [0.255, 0.650], p < 0.001).
  • Learning performance was moderated by type of course (Q_B = 64.249, p < 0.001; strongest for skills and competencies development, g = 0.874), learning model (Q_B = 76.220, p < 0.001; strongest for problem-based learning, g = 1.113, weakest for project-based learning, g = 0.239), and duration (Q_B = 55.998, p < 0.001; strongest at 4–8 weeks, g = 0.999). Grade level, role of ChatGPT, and application area were not significant.
  • Learning perception was moderated only by duration (Q_B = 19.839, p < 0.001), with >8 weeks showing the largest effect (g = 1.054) versus ≤1 week (g = 0.203).
  • Higher-order thinking was moderated by type of course (Q_B = 7.811, p < 0.05; STEM g = 0.737, language/writing g = 0.334, skills development g = 0.296) and role of ChatGPT (Q_B = 4.872, p < 0.05; intelligent tutor g = 0.945).

Methods

  • Design: meta-analysis following PRISMA guidelines, with literature search, inclusion/exclusion screening, coding of study characteristics, and data analysis; search covered Web of Science Core Collection, ScienceDirect, Taylor & Francis, Wiley Online Library, and Scopus, restricted to November 2022–February 2025, starting from 6,621 records and yielding 51 included articles.
  • Sample: 51 (quasi-)experimental studies with an experimental group using ChatGPT and a control group without it, both using the same learning method; 72 independent effect sizes were extracted. Interrater reliability for study selection was Cronbach's α = 0.96.
  • Analysis: Hedges's g computed in Comprehensive Meta-Analysis (3.0/3.3.0), interpreted via Sawilowsky's (2009) thresholds; random-effects models were used given heterogeneity (I² = 89.243% for learning performance, 72.580% for learning perception, 55.813% for higher-order thinking); publication bias assessed via funnel plots and fail-safe N (5490, 399, and 101 respectively), with one-study-removal sensitivity analysis.

Limitations

  • Stated by the authors: the sample of 51 documents included only one primary-school study and none at kindergarten level; only 19 and 9 studies examined learning perception and higher-order thinking respectively, versus a much larger sample for learning performance; only seven moderating variables were explored, omitting factors such as cultural background, parents' occupations, and national policies; and only experimental/quasi-experimental studies were included, excluding qualitative and mixed-methods research.
  • Evident from the text: heterogeneity was high for learning performance (I² = 89.243%), and the authors note some funnel-plot points fell outside the diagonal lines, indicating potential bias or heterogeneity; several moderator categories (e.g., mixed roles, intelligent partners for higher-order thinking, primary school) were excluded from analyses due to insufficient data.
  • Evident from the bibliographic record: the article is labeled "RETRACTED ARTICLE" in the title, though the retraction is not discussed in the full text provided.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1057/s41599-025-04787-yThis paper is marked retracted.

Cites

134 of 135 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 28 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The impact of ChatGPT on learning perception is moderated by duration (Q_B = 19.839, P < 0.001).

    “The impact of ChatGPT on learning performance was moderated by type of course (Q B = 64.249,P < 0.001), learning model (Q B = 76.220,P < 0.001), and duration (Q B = 55.998,P < 0.001); its effect on learning perception was moderated by duration (Q B = 19.839,P < 0.001); and its influence on the development of higher-order thinking was moderated by type of course (Q B = 7.811, P < 0.05) and the role”
    Quote in the paperSubstantial claim
  • Findingown data

    The impact of ChatGPT on learning performance is moderated by type of course (Q_B = 64.249, P < 0.001).

    “The impact of ChatGPT on learning performance was moderated by type of course (Q B = 64.249,P < 0.001), learning model (Q B = 76.220,P < 0.001), and duration (Q B = 55.998,P < 0.001); its effect on learning perception was moderated by duration (Q B = 19.839,P < 0.001); and its influence on the development of higher-order thinking was moderated by type of course (Q B = 7.811, P < 0.05) and the role”
    Quote in the paperSubstantial claim
  • Findingown data

    ChatGPT has a moderate positive effect on students' learning perception (g = 0.456).

    “The results indicate that ChatGPT has a large positive impact on improving learning performance (g = 0.867) and a moderately positive impact on enhancing learning perception (g = 0.456) and fostering higher-order thinking (g = 0.457).”
    • Also stated in: § Results · hedged
      “The overall effects of ChatGPT on enhancing learning performance, improving learning perception, and promoting higher-order thinking were calculated as g = 0.867, g = 0.456, and g = 0.457, respectively, indicating that ChatGPT has a large positive effect on learning performance and a medium positive effect on learning perception and higher-order thinking.”
    Quote in the paperSubstantial claim
  • Findingown data

    There is moderate heterogeneity among included studies regarding the effect of ChatGPT on higher-order thinking (Q = 18.105, p < 0.1, I2 = 55.813%).

    “The Q-test for higher-order thinking is 18.105, and the p-value of the Q-test is less than 0.1.The I² value is 55.813%, indicating moderate heterogeneity in the included studies regarding the effect of ChatGPT on higher-order thinking.”
    Quote in the paperSubstantial claim
  • Findingown data

    The impact of ChatGPT on learning performance is moderated by duration (Q_B = 55.998, P < 0.001).

    “The impact of ChatGPT on learning performance was moderated by type of course (Q B = 64.249,P < 0.001), learning model (Q B = 76.220,P < 0.001), and duration (Q B = 55.998,P < 0.001); its effect on learning perception was moderated by duration (Q B = 19.839,P < 0.001); and its influence on the development of higher-order thinking was moderated by type of course (Q B = 7.811, P < 0.05) and the role”
    Quote in the paperSubstantial claim
  • May combine two claims. 30
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 14

  • May combine two claims.

    Findingown data

    For learning perception, interventions lasting more than 8 weeks (g = 1.054) showed a substantially larger effect size than those lasting less than 1 week (g = 0.203), 1-4 weeks (g = 0.404), and 4-8 weeks (g = 0.493).

    “ChatGPT was found to be beneficial across all time periods, with interventions lasting more than 8 weeks (g = 1.054) showing a substantially larger effect size than those lasting less than 1 week (g = 0.203), 1-4 weeks (g = 0.404), and 4-8 weeks (g = 0.493).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ChatGPT has a substantial positive effect on students' learning performance, with an overall effect size of 0.867.

    “The results indicate that ChatGPT has a substantial positive effect on students' learning performance, with an overall effect size of 0.867.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Compared to its large positive impact on improving learning performance (g = 0.867), ChatGPT has only a moderately positive impact on learning perception.

    “Notably, compared to its large positive impact on improving learning performance (g = 0.867), ChatGPT has only a moderately positive impact on learning perception.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The improvements in learning performance attributed to ChatGPT are quite strong (g = 0.867), while the improvements in higher-order thinking are only moderate.

    “However, the improvements in learning performance attributed to ChatGPT are quite strong (g = 0.867), while the improvements in higher-order thinking are only moderate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The final sample consisted of 51 articles published between November 2022 and February 2025.

    “The final sample consisted of 51 articles, published between November 2022 and February 2025.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    In project-based learning, the effect of ChatGPT on students' learning performance is the weakest, showing only a small positive impact.

    “this study yielded a unique conclusion: in project-based learning, the effect of ChatGPT on students' learning performance is the weakest, showing only a small positive impact.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    For learning perception, among mixed role of ChatGPT, the mean effect size was g=0.313 (95% CI [0.024, 0.602]), Z=2.122, p<0.001 (as indicated by ***).

    “| Mixed | 2 | 0.313[0.024,0.602] | 2.122*** | 1 | | |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Sensitivity analysis showed that after removal of any single study, the 95% CI of the effect size for learning performance remained between 0.462 and 0.574 (fixed-effects model) and between 0.683 and 1.052 (random-effects model).

    “The results show that, for learning performance, after the removal of any single study, the range of the 95% confidence interval (CI) of the effect size still remained between 0.462 and 0.574 for the fixedeffects model and between 0.683 and 1.052 for the random-effects model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductcitation

    The fail-safe N for learning performance was 5490, which is greater than the tolerance value of 230, indicating that the possibility of publication bias is relatively small.

    “In this study, the fail-safe N for learning performance was calculated to be 5490 (5k + 10 = 230, k = 44), for learning perception was calculated to be 399 (5k + 10 = 105, k = 19), and for higher-order thinking was calculated to be 101 (5k + 10 = 55, k = 9), indicating that the possibility of publication bias is relatively small for this study.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The fail-safe N for learning perception was 399, greater than the tolerance value of 105, indicating that the possibility of publication bias is relatively small.

    “In this study, the fail-safe N for learning performance was calculated to be 5490 (5k + 10 = 230, k = 44), for learning perception was calculated to be 399 (5k + 10 = 105, k = 19), and for higher-order thinking was calculated to be 101 (5k + 10 = 55, k = 9), indicating that the possibility of publication bias is relatively small for this study.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The fail-safe N for higher-order thinking was 101, greater than the tolerance value of 55, indicating that the possibility of publication bias is relatively small.

    “In this study, the fail-safe N for learning performance was calculated to be 5490 (5k + 10 = 230, k = 44), for learning perception was calculated to be 399 (5k + 10 = 105, k = 19), and for higher-order thinking was calculated to be 101 (5k + 10 = 55, k = 9), indicating that the possibility of publication bias is relatively small for this study.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The funnel plots for the effects of ChatGPT on learning performance, learning perception, and higher-order thinking are relatively evenly distributed on both sides, indicating sufficiently high reliability for meta-analysis.

    “The effect values for the impact of ChatGPT on learning performance, learning perception, and higher-order thinking are relatively evenly distributed on both sides of the funnel plot, indicating that the data have sufficiently high reliability to be used for meta-analysis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    All relevant studies for higher-order thinking were conducted at the college level, so no grade-level moderator analysis was performed.

    “All relevant studies were conducted at the college level, and thus no analysis was performed for grade level.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The experimental and control groups had to adopt the same learning method.

    “Furthermore, the experimental and control groups had to adopt the same learning method.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    ChatGPT can make a course more interesting and encourage students to actively search for information, ultimately contributing to an increase in learning perception.

    “ChatGPT can make a course more interesting and encourage students to actively search for information, ultimately contributing to an increase in learning perception.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The literature inclusion and exclusion criteria were developed by two researchers specializing in educational technology.

    “The literature inclusion and exclusion criteria were developed by two researchers specializing in educational technology, and are presented in Table 1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    This finding agrees with the results of Guan et al. (2024) and may be attributed to the consistent positive feedback students receive from ChatGPT during extended usage.

    “This finding agrees with the results of Guan et al. (2024) and may be attributed to the consistent positive feedback students receive from ChatGPT during extended usage.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Guan et al. 2024 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The various roles ChatGPT can play and the areas in which it can be applied are essentially manifestations of its core function as a learning support tool.

    “The various roles it can play and the areas in which it can be applied are essentially manifestations of its core function as a learning support tool.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation

    This may be because such courses typically involve well-defined task objectives and procedural steps; ChatGPT provides immediate feedback, targeted guidance, and problem-solving support, and thus can enable students to complete such tasks more efficiently.

    “This may be because such courses typically involve well-defined task objectives and procedural steps; ChatGPT provides immediate feedback, targeted guidance, and problemsolving support (Rakap, 2023;Urban et al. 2024), and thus can enable students to complete such tasks more efficiently.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Rakap, 2023

      Not checked in the free run.

    • cites Urban et al. 2024

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The primary school sample was excluded from the learning performance moderator analysis because it consisted of only one study, which did not find a statistically significant effect.

    “It should be noted that the primary school sample was excluded from the analysis because it consisted of only one study, which did not find a statistically significant effect.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    ChatGPT-generated content is not always completely accurate, and this bias can hinder the development of students' higher-order thinking, thus limiting the overall degree to which ChatGPT can promote students' higher-order thinking.

    “Moreover, ChatGPT-generated content is not always completely accurate, and this bias can hinder the development of students' higherorder thinking (Niloy et al. 2023), thus limiting the overall degree to which ChatGPT can promote students' higher-order thinking.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Niloy et al. 2023

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    ChatGPT may lack critical analysis abilities and the ability to provide creative problem-solving solutions.

    “It thus may lack critical analysis abilities and the ability to provide creative problem-solving solutions (Lu et al. 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lu et al. 2024

      Not checked in the free run.

    • Rests on an assumption: ChatGPT is based on pretrained models that rely on existing data and patterns.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation

    Prolonged use of ChatGPT can lead students to become overly reliant on the AI tool, and thus they may neglect to reinforce the knowledge they have learned, which, in turn, would result in a decline in learning performance.

    “This may be because the prolonged use of ChatGPT can lead students to become overly reliant on the AI tool, and thus they may neglect to reinforce the knowledge they have learned (Agarwal, 2023), which, in turn, would result in a decline in learning performance.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Agarwal, 2023

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    These features enable ChatGPT to support more diverse learning scenarios, which may, in turn, contribute to its larger effect size.

    “These features enable ChatGPT to support more diverse learning scenarios, which may, in turn, contribute to its larger effect size.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation

    This may be because project-based learning emphasizes the completion of a comprehensive project in a real-world context; in such a scenario, ChatGPT's support for addressing complex problem-solving tasks may be limited, resulting in only a minor effect.

    “This may be because project-based learning emphasizes the completion of a comprehensive project in a realworld context (Botha, 2010).In such a scenario, ChatGPT's support for addressing complex problem-solving tasks may be limited, resulting in only a minor effect.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Botha, 2010

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Hedges's g was used to adjust effect sizes to correct for small sample size biases.

    “Since differences in sample sizes between studies might introduce bias into the calculated effect sizes, Hedges's g was used to adjust the effect sizes to correct for small sample size biases (Hedges, 1981).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hedges, 1981 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    This may be attributed to its inherent strengths, in that ChatGPT excels at providing clear explanations for a wide range of problems, whether establishing simple correspondences between geometric structures and algebraic equations or explaining complex scientific concepts.

    “This may be attributed to its inherent strengths, in that ChatGPT excels at providing clear explanations for a wide range of problems, whether establishing simple correspondences between geometric structures and algebraic equations (Poola and Božić, 2023) or explaining complex scientific concepts (Ali et al. 2024).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Poola and Božić, 2023

      Not checked in the free run.

    • cites Ali et al. 2024 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    Meta-analysis can be used to comprehensively analyze the results of multiple independent research studies focused on the same research purpose, reveal differences and commonalities among them, and draw universal and regular conclusions.

    “Meta-analysis can be used to comprehensively analyze the results of multiple independent research studies that are focused on the same research purpose, reveal the differences and commonalities among them, and thereby draw universal and regular conclusions (Lipsey and Wilson, 2001).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lipsey and Wilson, 2001

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    In STEM and related courses, students are often required to collaborate with ChatGPT to complete complex projects, and the process of completing such projects inherently promotes the development of higher-order thinking skills such as creative thinking, problem-solving, and critical thinking.

    “In STEM and related courses, students are often required to collaborate with ChatGPT to complete complex projects (Li, 2023;Li et al. 2024), and the process of completing such projects inherently promotes the development of higher-order thinking skills such as creative thinking, problem-solving, and critical thinking.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Li, 2023

      Not checked in the free run.

    • cites Li et al. 2024

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This may be due to the fact that when students first started using ChatGPT, they lacked the questioning skills needed to use it effectively.

    “This may be due to the fact that when students first started using ChatGPT, they lacked the questioning skills needed to use it effectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Articles that met the coding criteria were independently selected by two researchers, and acceptable interrater reliability was achieved (Cronbach's α = 0.96).

    “Articles that met the coding criteria were independently selected by two researchers, and acceptable interrater reliability was achieved (Cronbach's α = 0.96).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    When using ChatGPT, students continuously reflect on their learning processes and adjust their learning strategies in real-time; this process effectively fosters the development of students' higher-order thinking skills.

    “When using ChatGPT, students continuously reflect on their learning processes and adjust their learning strategies in real-time.This process effectively fosters the development of students' higherorder thinking skills (Lu et al. 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lu et al. 2024

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given

    This study only selected experimental and quasi-experimental studies and did not include relevant qualitative or mixed-methods research.

    “Finally, this study only selected experimental and quasi-experimental studies and did not include relevant qualitative or mixed-methods research.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    This study's sample size was limited, comprising only 51 documents, of which only one addressed primary school and none addressed kindergarten.

    “First, this study sample size was limited, comprising only 51 documents, of which only one addressed primary school and none addressed kindergarten.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    This study only explored seven moderating variables, but there are many more variables that could potentially affect the impact of ChatGPT on student learning, such as students' cultural backgrounds, parents' occupations, and relevant national policies.

    “Second, this study only explored seven moderating variables, but there are many more variables that could potentially affect the impact of ChatGPT on student learning, such as students' cultural backgrounds, parents' occupations, and relevant national policies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The number of included samples that examined learning perception and higher-order thinking was relatively small, with only 19 and 9 studies, respectively, compared to a significantly larger sample for learning performance; this imbalance may limit the ability to comprehensively understand ChatGPT's impact on learning perception and higher-order thinking.

    “Moreover, the number of included samples that examined learning perception and higher-order thinking was relatively small, with only 19 and 9 studies, respectively, compared to a significantly larger sample for learning performance.This imbalance may limit our ability to comprehensively understand ChatGPT's impact on learning perception and higher-order thinking.”
    Quote in the paperSubstantial claim
36 of 36

  • The paper relies on this without support.

    ChatGPT is based on pretrained models that rely on existing data and patterns.

    “ChatGPT is based on pretrained models that rely on existing data and patterns.”

  • The paper relies on this without support.

    The meta-analysis by Chen et al. (2025) on other AI assessment tools is comparable to this meta-analysis on ChatGPT.

    “The results indicate that the effect size for the impact of ChatGPT on students' learning performance is significantly larger than that of other AI assessment tools (g = 0.390).”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Wang and Fan (2025) found a large effect on learning performance, particularly when structured frameworks like Bloom's Taxonomy are used to guide AI-supported tasks.

    “Their study also found a large effect on learning performance, particularly when structured frameworks like Bloom's Taxonomy are used to guide AI-supported tasks.”

    From Enhancing service-learning through Generative AI: a mixed-methods study on educational Game design in a finance course (2025) · cites this paper as Wang and Fan (2025)

    The Abstract and the section “How effective is ChatGPT in promoting learning performance?” report a large positive effect on learning performance (g = 0.867). The Abstract recommends frameworks such as Bloom’s taxonomy for developing higher-order thinking, not as a condition under which the learning-performance effect was particularly large. Our copy provides no page numbers for these passages. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • Wang and Fan (2025) found that ChatGPT has a moderate effect on higher-order thinking, but only when appropriate scaffolding is present.

    “Wang and Fan (2025) found that ChatGPT has a moderate effect on higher-order thinking, but only when appropriate scaffolding is present, such as when instructors position ChatGPT as a learning partner.”

    From Enhancing service-learning through Generative AI: a mixed-methods study on educational Game design in a finance course (2025) · cites this paper as Wang and Fan (2025)

    The Results report a moderate positive effect on higher-order thinking (g = 0.457), and the Conclusion and implications recommends learning scaffolds to improve effectiveness. Neither establishes that the effect occurs only with scaffolding. The Results also say the learning-partner role was excluded from the role analysis because only one study represented it, so the cited finding does not specifically establish the instructor-positioned learning-partner example. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • Much of the literature on AI-supported feedback has focused on general writing tasks or broad comparisons among feedback sources, whereas disciplinary reasoning in university science courses has received less sustained attention.

    “much of the literature on AI-supported feedback has focused on general writing tasks or broad comparisons among feedback sources, whereas disciplinary reasoning in university science courses has received less sustained attention”

    From Human-centered GenAI feedback design in higher education: a multisite experiment on direct, reflective, and hybrid approaches to scientific argumentation (2026) · cites this paper as Wang & Fan, 2025

    The Literature review (page unavailable) describes studies of ChatGPT-assisted text revision and writing, supporting the claim that writing tasks have received attention. It also describes research in STEM subjects and does not establish that feedback-source comparisons predominate or that disciplinary reasoning in university science courses has received less sustained attention. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

  • Wang and Fan (2025) analysed studies from a similar period and reported a substantial positive effect of ChatGPT on learning, comparable to Deng et al.

    “Wang and Fan (2025) analysed studies from a similar period and reported comparable findings to those of Deng and colleagues: a substantial positive effect of ChatGPT on learning.”

    From <scp>ChatGPT</scp> in Education: An Effect in Search of a Cause (2025) · cites this paper as Wang and Fan (2025)

  • GenAI broadens forms of expression, strengthens creativity, and supports higher-order and critical thinking.

    “which broadens forms of expression, strengthens creativity [4,5], and supports higher-order and critical thinking [6,7].”

    From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Number of moderating variables changes from six to seven

    Minor

    The methods specify six selected moderating variables, but the limitations section says the study explored seven.

  • Over-eight-week confidence interval is not centered on its effect estimate

    Minor

    The midpoint of the reported 95% confidence interval is 0.587, not the stated effect estimate of 0.531.

    Reported 0.531 · recomputed 0.587

  • Language-course confidence interval is not centered on its effect estimate

    Minor

    The reported effect estimate equals the upper endpoint of its 95% confidence interval. The midpoint of the stated endpoints is 0.4205, rather than 0.531.

    Reported 0.531 · recomputed 0.4205

  • Data-availability statement conflicts with the reported meta-analysis

    Minor

    The paper says no datasets were analysed, although it reports extracting and analysing 72 effect sizes from 51 articles.

  • Learning-performance Q statistic differs between text and table

    Minor

    The heterogeneity discussion gives Q as 339.725, whereas Table 4 gives 399.725 for the same outcome.

5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Alsharif MHK, Elamin AY, Almasaad JM, Bakhit NM, Alarifi A, Taha KM, Hassan WA, Zumrawi E (2024) Using ChatGPT to create engaging problem-based learning scenarios in anatomy: a step-by-step guide. New Armen Med J 18(4). https://doi.org/10.56936/18290825-4.v18.2024-98 · DOI
  2. Lipsey MW, Wilson DB (2001) Practical meta-analysis. Sage Publications, Inc
  3. Luo H, Liao X, Ru Q, Wang Z (2024) Generative AI-supported teacher comments: an empirical study based on junior high school mathematics classrooms. e-Educ Res 45(5):58–66. https://doi.org/10.13811/j.cnki.eer.2024.05.008 · DOI
  4. Nugroho A, Putro N, Syamsi K (2023) The potentials of ChatGPT for language learning: unpacking its benefits and limitations. Register J 16(2):224–247. https://doi.org/10.18326/rgt.v16i2.224-247 · DOI
  5. Poola I, Božic V (2023) Guiding AI with human intuition for solving mathematical problems in ChatGPT. Int J Eng Sci Res 11(07):31–40. https://www.researchgate.net/publication/373447147
  6. Vygotskij LS, Hanfmann E (1966) Thought and language. MIT Press

Cited by

Sorted by citations

Related

Authors
Wang, J., & Fan, W.
Affiliations
Hangzhou Normal University (CN)
Corresponding author
Fan, Wenxiang
ORCID
Venue
Humanities and Social Sciences Communications
Publisher
Springer Science and Business Media LLC
Published
2025-05-06
Volume
12
Issue
1
ISSN
  • 2055-1045 (ISSN-L)
  • 2662-9992
DOI
10.1057/s41599-025-04787-y
OpenAlex
W4410129808
Semantic Scholar
45606f671465c499d9299234e8c5a3f444556f1b
Cited by
322
References
134
Citations by year
2026: 217 · 2025: 105
FWCI
48.98
Type
journal article
Language
en
Status
Retracted
Update notices
retraction (2026-04-22)
Open access
Licence
Locations
  • Humanities and Social Sciences Communications (publishedVersion, cc-by-nc-nd)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Major Project of Humanities and Social Sciences Key Research Base of the Ministry of Educatio (22JJD880023)
  • University-Industry Collaborative Education Program (240904171111322)
  • Hangzhou Social Science Routine Planning Project (Z24JC014)
  • Hangzhou Normal University’s 2023 Teaching Construction and Reform Project “Exploration and Practice of Generative Artificial Intelligence to Promote University Teaching Reform”
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
ChatGPT, generative AI in education, learning performance, perceptions of learning, higher-order thinking skills, AI-assisted learning, problem-based learning, Bloom's taxonomy, instructional scaffolding, intelligent tutoring systems, moderating effect

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review