CoChat
Sign Up
Cited by 48Artificial Intelligence in EducationArtificial Intelligence in Healthcare and EducationEducation and Critical Thinking Development

Investigating the Use of chatGPT as a Tool for Enhancing Critical Thinking and Argumentation Skills in International Relations Debates among Undergraduate Students

Open access
de la Puente, Mario, Torres, Jose, Troncoso, Ana Laura Blanco, Meza, Yuraima Yuliza Hernández, Carrascal, Jenny Xiomara Marquez·2024·Smart Learning Environments
DOI 10.1186/s40561-024-00347-0
Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract This study investigated the effectiveness of using ChatGPT, a large language model (LLM), to enhance critical thinking and argumentation skills among undergraduate students studying international relations in a developing nation context. A total of 95 participants were randomly assigned to an experimental group ( n = 48) and a control group ( n = 47). The experimental group engaged in debate sessions using ChatGPT, while the control group participated in traditional debate sessions. Pre-test and post-test assessments were conducted to measure the participants’ critical thinking and argumentation skills. The results revealed that the use of ChatGPT significantly improved students' understanding of complex concepts, critical thinking, and argumentation skills compared to traditional debate methods. The findings suggest that integrating LLMs into existing educational curricula may be an effective strategy for enhancing students’ critical thinking and argumentation skills in resource-constrained settings. However, the challenges and limitations associated with the use of LLMs in developing nations should be carefully considered. The study contributes to the growing body of research on the application of AI-powered tools in education and informs the development of evidence-based practices and policies for integrating LLMs into educational curricula.

AI snapshot

Written from the paper’s full text.

TL;DR

This randomized pre–post study found that using ChatGPT in international relations debate sessions improved undergraduates’ understanding of complex concepts, critical thinking, and argumentation skills compared with traditional debates. The authors report moderate effects and argue that LLMs may help resource-constrained educational settings, while cautioning about bias, misinformation, and infrastructure limits.

Key findings

  • The experimental group outperformed the control group on the combined post-test outcomes after controlling for pre-test scores: MANCOVA Wilks’ Λ = 0.76, F(3, 88) = 9.12, p < 0.001, partial η² = 0.24.
  • ANCOVAs showed significant ChatGPT effects on all three outcomes: complex concepts F(1, 90) = 12.68, p < 0.001, partial η² = 0.12; critical thinking F(1, 90) = 15.37, p < 0.001, partial η² = 0.15; argumentation F(1, 90) = 18.92, p < 0.001, partial η² = 0.17.
  • SEM found direct effects of ChatGPT on critical thinking (β = 0.48, p < 0.001, 95% CI 0.28–0.68) and argumentation skills (β = 0.55, p < 0.001, 95% CI 0.37–0.73), plus a significant indirect effect on understanding complex concepts (β = 0.45, p < 0.001, 95% CI 0.27–0.63).
  • Pre-test scores did not differ significantly between groups: experimental mean = 68.5 (SD = 10.3), control mean = 67.2 (SD = 9.8), t(93) = 0.78, p = 0.44.

Methods

  • Randomized experimental design with 95 undergraduate international relations students at a private university in Colombia; participants were assigned to an experimental group (n = 48) using ChatGPT in debate sessions or a control group (n = 47) using traditional debate sessions.
  • The intervention ran over a one-year period in 2023, with weekly debate sessions lasting approximately 90 minutes; all participants completed pre- and post-tests (100 points, 90 minutes) measuring knowledge, critical thinking, and argumentation, plus open-ended qualitative questions in the post-test.
  • Analyses included independent-samples t-test on pre-test scores, MANCOVA and follow-up ANCOVAs controlling for pre-test scores, and structural equation modeling (SEM) to examine direct and indirect relationships among ChatGPT use, critical thinking, argumentation, and understanding of complex concepts.

Limitations

  • The paper states that the relatively small sample size and focus on a specific international relations student population may limit generalizability to other contexts and disciplines, and that it did not explore long-term effects of LLM use on critical thinking.
  • The paper also states that challenges in developing nations—potential perpetuation of biases and misinformation, and limited access to reliable internet connectivity and computing devices—should be carefully considered when interpreting results.
  • Evident from the text: the study was conducted under a single professor at one private university in Colombia, which further narrows the institutional and geographic scope beyond the authors’ stated generalizability concern.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s40561-024-00347-0

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

13 of 23 cited sources checked · 4 supported · 2 flagged · accuracy not yet measured

  • Interpretationcitation

    The findings support the notion that LLMs such as ChatGPT have the potential to enhance students' critical thinking skills, including understanding complex concepts, engagement in critical thinking, and construction of well-reasoned arguments.

    “The findings support the notion that LLMs, such as ChatGPT, have the potential to enhance students' critical thinking skills, including their understanding of complex concepts, engagement in critical thinking, and construction of well-reasoned arguments (Guo, 2023;Kasneci et al., 2023;Yang et al., 2022).”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Guo, 2023 source

      The Conclusions report improved student self-perception of critical-thinking competence, including comprehending complex subject matter, and describe ChatGPT’s potential to develop those skills. The Examples of Students’ Essays describe students finding evidence to strengthen essays, but the paper does not establish that ChatGPT enhanced their construction of well-reasoned arguments; the reported improvement is primarily self-perceived. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Kasneci et al., 2023 source

      Section 1.1 (page unavailable) says LLMs can help students develop critical thinking and understand difficult texts through explanations and prompts. But this is a discussion of potential uses, not a finding that students' skills improved; it does not specifically establish that LLMs help students construct well-reasoned arguments. Section 4 (page unavailable) also cautions that overreliance may harm critical thinking. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Yang et al., 2022 source

      The Results and Discussion (pages unavailable in our copy) evaluate GatorTron on clinical language-processing tasks, including medical question answering. Searches for students, education, ChatGPT, and critical thinking found no passage addressing whether LLMs improve students’ understanding, critical thinking, or argument construction. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data

    The use of ChatGPT significantly improved students' argumentation skills compared to traditional debate methods.

    “The results revealed that the use of ChatGPT significantly improved students' understanding of complex concepts, critical thinking, and argumentation skills compared to traditional debate methods.”
    • Also stated in: § Literature review · boosted
      “This finding is consistent with the current study's results, which demonstrated significant improvements in students' critical thinking and argumentation skills when using ChatGPT in debate sessions.”
    Quote in the paperSubstantial claim
  • Findingown data

    The use of ChatGPT significantly improved students' understanding of complex concepts compared to traditional debate methods.

    “The results revealed that the use of ChatGPT significantly improved students' understanding of complex concepts, critical thinking, and argumentation skills compared to traditional debate methods.”
    Quote in the paperSubstantial claim
  • Findingown data

    The use of ChatGPT significantly improved students' critical thinking compared to traditional debate methods.

    “The results revealed that the use of ChatGPT significantly improved students' understanding of complex concepts, critical thinking, and argumentation skills compared to traditional debate methods.”
    • Also stated in: § Literature review · boosted
      “This finding is consistent with the current study's results, which demonstrated significant improvements in students' critical thinking and argumentation skills when using ChatGPT in debate sessions.”
    Quote in the paperSubstantial claim
  • Findingown data

    ANCOVA showed that ChatGPT significantly improved students' critical thinking skills (F(1, 90) = 15.37, p < 0.001, partial η2 = 0.15).

    “critical thinking skills (F(1, 90) = 15.37,p < 0.001, partial η 2 = 0.15)”
    • Also stated in: § Conclusion · boosted
      “The results demonstrated improvements in students' understanding of complex concepts, critical thinking, and argumentation skills following their participation in Chat-GPT-based debate sessions compared to traditional debate methods.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The pre-test and post-test, as scored by the rubrics, validly measure the intended constructs of understanding complex concepts, critical thinking, and argumentation skills.
    • Rests on an assumption: The statistical assumptions of MANCOVA and SEM (e.g., normality, linearity, homoscedasticity) were satisfied.
  • Classified as probably not a substantial claim. 19
  • May combine two claims. 19

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLM-based critical thinking training programs significantly improved students' critical thinking abilities.

    “Suratmi and Sopandi (2022) reported significant improvements in the critical thinking abilities of Nigerian secondary school students who participated in an LLM-based critical thinking training program.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Suratmi & Sopandi, 2022 source

      The Research Method section describes a questionnaire study of elementary-school teachers, not a training program involving students. The Conclusion states that the study only describes teachers’ knowledge, skills, and attitudes and leaves implementation for future research. It therefore does not report significant improvements among Nigerian secondary-school students from an LLM-based program. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Interpretationcitation

    The development of critical thinking skills through the use of LLMs, as observed in this study, is consistent with the findings of Suratmi and Sopandi (2022) and Snyder and Wiles (2015), who reported significant improvements in students' critical thinking abilities following LLM-based training programs and collaborative learning interventions.

    “The development of critical thinking skills through the use of LLMs, as observed in this study, is consistent with the findings of Suratmi and Sopandi (2022) and Snyder and Wiles (2015), who reported significant improvements in students' critical thinking abilities following LLM-based training programs and collaborative learning interventions.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Suratmi and Sopandi (2022) source

      Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites Snyder and Wiles (2015) source

      Split into 2 atoms — 0 confirmed, 0 partially supported, 2 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    LLMs such as ChatGPT have shown promise in improving students' critical thinking disposition and their ability to construct well-reasoned arguments.

    “LLMs such as ChatGPT have shown promise in improving students' critical thinking disposition and their ability to construct well-reasoned arguments (Kasneci et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kasneci et al., 2023 source

      Section 1.1, “Opportunities for Learning” (page unavailable), says LLMs can assist students in developing critical-thinking skills and can support writing and debate. But it does not establish improvement in critical-thinking disposition or in students’ ability to construct well-reasoned arguments. Section 2.2 reports improved revisions of scientific arguments from an AI-based feedback system, not specifically from an LLM such as ChatGPT. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Guo (2023) concluded that ChatGPT could play a transformative role in education, offering new ways for students to develop critical thinking skills.

    “The study concluded that ChatGPT could play a transformative role in education, offering new ways for students to develop critical thinking skills.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Guo, 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Large transformer models such as ChatGPT exhibit significant improvements over previous NLP models in terms of emergence and homogenization, which could contribute to enhancing critical thinking skills in students.

    “Recent studies have demonstrated that large transformer models, such as ChatGPT, trained using massive text data, exhibit significant improvements over previous NLP models in terms of emergence and homogenization, which could contribute to enhancing critical thinking skills in students (Yang et al., 2022).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Yang et al., 2022 source

      The Discussion reports that GatorTron outperformed earlier biomedical and clinical transformer models on five clinical NLP tasks. The paper does not establish improvements in emergence or homogenization, evaluate ChatGPT, or address students’ critical thinking skills. The cited result supports only the broad point that a large transformer model improved on previous models in its tested tasks. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Kasneci et al. (2023) found that ChatGPT could significantly enhance students' learning outcomes, particularly in areas requiring higher-order thinking skills.

    “The results of Kasneci et al. 's study indicated that ChatGPT could significantly enhance students' learning outcomes, particularly in areas requiring higher-order thinking skills.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kasneci et al., 2023 source

      Section 1.1 (Opportunities for Learning; page unavailable) says large language models may improve student performance and help develop critical-thinking and problem-solving skills. It presents these as potential benefits, not as results of a Kasneci et al. study showing a significant improvement in learning outcomes, particularly for higher-order thinking skills. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Concerns have been raised that large language models may have biases and limitations, requiring careful and critical use.

    “Concerns have been raised regarding the potential biases and limitations of large language models, emphasizing the need for careful and critical use to address these issues (Bitzenbauer, 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Bitzenbauer, 2023 source

  • May combine two claims.

    Interpretationcitation

    The study's findings extend the perspective that leveraging ChatGPT enhances critical thinking to the specific domain of international relations, demonstrating that AI effectively challenges students' existing knowledge structures and encourages deeper analytical thinking in this field.

    “These findings extend this perspective to the specific domain of international relations, demonstrating that AI effectively challenges students' existing knowledge structures and encourages deeper analytical thinking in this field.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Guo (2023) source

  • May combine two claims.

    Interpretationcitation

    The findings support cognitive load theory, as discussed by Liang et al. (2023), suggesting that ChatGPT effectively reduces cognitive overload for students by presenting complex information in structured, digestible steps.

    “The findings support the cognitive load theory, as discussed by Liang et al. (2023), suggesting that ChatGPT effectively reduces cognitive overload for students by presenting complex information in structured, digestible steps.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Liang et al. (2023) source

  • May combine two claims.

    Findingown data

    The SEM model demonstrated good fit indices (CFI = 0.97, RMSEA = 0.04, SRMR = 0.03), indicating that the hypothesized relationships were well-supported by the data.

    “The model demonstrated good fit indices (CFI = 0.97, RMSEA = 0.04, SRMR = 0.03), indicating that the hypothesized relationships were well-supported by the data.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The statistical assumptions of MANCOVA and SEM (e.g., normality, linearity, homoscedasticity) were satisfied.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Use of LLMs or ChatGPT in social studies significantly improved students' comprehension of intricate political and economic concepts.

    “Furthermore, the use of LLMs in specific courses, such as social studies, has been found to significantly improve students' comprehension of intricate political and economic concepts (Xie, 2023).”
    • Also stated in: § Literature review · plain
      “Furthermore, Xie (2023) conducted a study in India and found that the use of Chat-GPT in a social studies course significantly improved students' comprehension of intricate political and economic concepts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Xie, 2023 source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The pre-test was developed by a panel of international relations experts and validated through pilot testing.

    “The pre-test consisted of multiple-choice questions, short answer questions, and essay prompts that were developed by a panel of international relations experts and validated through pilot testing.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Nigerian secondary school students who engaged in LLM-based argumentation training exhibited enhanced abilities to identify and evaluate the quality of evidence used in arguments.

    “Similarly, Nigerian secondary school students who engaged in LLM-based argumentation training exhibited enhanced abilities to identify and evaluate the quality of evidence used in arguments (Kuhn & Crowell, 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kuhn & Crowell, 2011 source

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Participants were randomly assigned to the experimental group (n = 48) or the control group (n = 47) to ensure an unbiased distribution.

    “The students were randomly assigned to either an experimental group (n = 48) or a control group (n = 47) to ensure an unbiased distribution of participants.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The results provide strong evidence supporting the hypothesis that the use of ChatGPT in debate sessions would enhance students' critical thinking, argumentation skills, and understanding of complex concepts in international relations.

    “The results of the MANCOVA, ANCOVAs, and SEM analyses provide strong evidence supporting the hypothesis that the use of ChatGPT in debate sessions would enhance students' critical thinking, argumentation skills, and understanding of complex concepts in international relations.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The experimental and control groups were equivalent on all relevant variables other than the ChatGPT intervention, so the observed post-test differences can be attributed to ChatGPT.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study did not explore the long-term effects of using LLMs on students' critical thinking skills.

    “Additionally, the study did not explore the long-term effects of using LLMs on students' critical thinking skills, which is an important consideration for future research.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Overreliance on AI-generated responses could lead to a decrease in independent thinking if students uncritically accept the AI's outputs.

    “There is a risk that overreliance on AI-generated responses could lead to a decrease in independent thinking if students uncritically accept the AI's outputs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Inter-rater reliability of scoring was assessed using Cohen's kappa, and discrepancies in scores were resolved through discussion and consensus between the raters.

    “The inter-rater reliability was assessed using Cohen's kappa, and any discrepancies in scores were resolved through discussion and consensus between the raters (Appendix 1).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Engaging in argumentation leads to a more secure understanding of pre-existing concepts and allows students to hear new ideas, which can eliminate misconceptions.

    “In addition, the study by Venville and Dawson (2010) supports the idea that engaging in argumentation leads to a more secure understanding of pre-existing concepts and allows students to hear new ideas, which can eliminate misconceptions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Venville & Dawson, 2010 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The study highlights the potential of LLMs to facilitate personalized learning experiences and improve students' understanding of complex concepts, which is particularly relevant in resource-constrained settings.

    “The study also highlights the potential of LLMs to facilitate personalized learning experiences and improve students' understanding of complex concepts, which is particularly relevant in resource-constrained settings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The use of ChatGPT had a significant impact on students' learning outcomes, with the experimental group outperforming the control group on post-test measures.

    “This finding suggests that the use of ChatGPT had a significant impact on students' learning outcomes, with the experimental group outperforming the control group on the post-test measures.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The experimental and control groups were equivalent on all relevant variables other than the ChatGPT intervention, so the observed post-test differences can be attributed to ChatGPT.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The relatively small sample size and the focus on a specific student population within international relations may limit the generalizability of the findings to other contexts and disciplines.

    “The relatively small sample size and the focus on a specific student population within international relations may limit the generalizability of the findings to other contexts and disciplines.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    ChatGPT had a moderate impact on each learning outcome, with effect sizes ranging from 0.12 to 0.17.

    “The effect sizes ranged from 0.12 to 0.17, indicating that ChatGPT had a moderate impact on each learning outcome.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationown data

    The findings align with the existing literature on the use of LLMs in education, particularly in the context of developing nations, highlighting the potential of these tools to facilitate personalized learning experiences, improve students' comprehension of intricate ideas, and foster the development of essential skills such as critical thinking and argumentation.

    “The findings align with the existing literature on the use of LLMs in education, particularly in the context of developing nations, highlighting the potential of these tools to facilitate personalized learning experiences, improve students' comprehension of intricate ideas, and foster the development of essential skills such as critical thinking and argumentation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Integrating ChatGPT into debate sessions can significantly improve critical thinking, argumentation, and understanding of complex concepts, thereby better preparing students for the challenges and demands of the field.

    “The results suggest that integrating ChatGPT into debate sessions can significantly improve these essential skills, thereby better preparing students for the challenges and demands of the field.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The findings provide evidence for the effectiveness of ChatGPT in enhancing students' critical thinking and argumentation skills, which are essential for success in various academic and professional contexts.

    “The findings provide evidence for the effectiveness of ChatGPT in enhancing students' critical thinking and argumentation skills, which are essential for success in various academic and professional contexts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    ChatGPT has the potential to stimulate critical thinking by presenting students with diverse perspectives and challenging their existing knowledge structures.

    “ChatGPT, as an advanced language model, has the potential to stimulate critical thinking by presenting students with diverse perspectives and challenging their existing knowledge structures.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The structured debate sessions and guided interactions with ChatGPT described in this study present a practical model that educators can adapt and implement in their own classrooms.

    “The structured debate sessions and guided interactions with ChatGPT described in this study present a practical model that educators can adapt and implement in their own classrooms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    As students engaged with ChatGPT, they not only improved their critical thinking and argumentation skills but also developed their ability to interact effectively with AI technologies.

    “As students engaged with ChatGPT, they not only improved their critical thinking and argumentation skills but also developed their ability to interact effectively with AI technologies.”
    Quote in the paperSubstantial claim
29 of 29

  • The paper relies on this without support.

    Existing research on LLMs in education has been predominantly conducted in developed countries, leaving a significant gap in knowledge about their applicability and effectiveness in resource-constrained settings.

    “Existing research on LLMs in education has been predominantly conducted in developed countries, leaving a significant gap in knowledge about their applicability and effectiveness in resource-constrained settings.”

  • The paper relies on this without support.

    Current empirical evidence on the long-term effects of using LLMs on students' critical thinking and argumentation skills, especially in international relations, remains limited.

    “Current empirical evidence on the long-term effects of using LLMs on students' critical thinking and argumentation skills, especially in the field of international relations, remains limited.”

  • The paper relies on this without support.

    The integration of LLMs like ChatGPT into existing educational curricula in resource-constrained settings presents unique challenges that have not been thoroughly addressed in the literature.

    “The integration of LLMs like ChatGPT into existing educational curricula in resourceconstrained settings presents unique challenges that have not been thoroughly addressed in the literature.”

  • The paper relies on this without support.

    There is a gap in understanding how LLMs can be specifically tailored to enhance critical thinking and argumentation skills within international relations.

    “Moreover, there is a gap in understanding how LLMs can be specifically tailored to enhance critical thinking and argumentation skills within the complex and rapidly evolving field of international relations.”

  • The paper relies on this without support.

    Limited access to reliable internet connectivity and computing devices in some developing nations may limit the widespread adoption of LLMs in education.

    “Additionally, the limited access to reliable internet connectivity and computing devices in some developing nations may limit the widespread adoption of LLMs in education.”

  • The paper relies on this without support.

    The pre-test and post-test, as scored by the rubrics, validly measure the intended constructs of understanding complex concepts, critical thinking, and argumentation skills.

    “The rubrics were developed by the expert panel and included criteria such as the accuracy and relevance of the content provided by Education for All Online E.U organization, an independent entity from the university were the study was performed.”

  • The paper relies on this without support.

    The experimental and control groups were equivalent on all relevant variables other than the ChatGPT intervention, so the observed post-test differences can be attributed to ChatGPT.

    “The pre-test scores served as covariates to control for any initial differences between the groups.”

  • The paper relies on this without support.

    The statistical assumptions of MANCOVA and SEM (e.g., normality, linearity, homoscedasticity) were satisfied.

    “Inferential statistics, specifically multivariate analysis of covariance (MANCOVA) and structural equation modeling (SEM), were employed to examine the impact of ChatGPT on students' learning outcomes.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI agents assumed roles such as peer reviewer, expert stakeholder, regulator, or devil's advocate to challenge students' reasoning and surface underlying assumptions.

    “In these designs, AI agents assumed roles such as peer reviewer, expert stake holder, regulator, or devil's advocate to challenge students' reasoning and surface underlying assumptions [56,76,93].”

    From A Systematic Literature Review on the Pedagogical Implications and Impact of GenAI on Students' Critical Thinking (2026) · cites this paper as [56]

    The Research method describes ChatGPT playing devil’s advocate, generating counterarguments, and helping students give peers feedback on their arguments. It does not describe the AI assuming expert-stakeholder or regulator roles, and the peer-review activity involved students reviewing classmates’ work with ChatGPT’s help rather than the AI explicitly acting as a peer reviewer. The passages have no page numbers in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentt(93) = 0.78reported p = .44 · recomputed p = .437

    “t(93) = 0.78, p = .44”

  • ConsistentF(3, 88) = 9.12reported p < .001 · recomputed p < .001

    “F(3, 88) = 9.12, p < .001”

  • ConsistentF(1, 90) = 12.68reported p < .001 · recomputed p < .001

    “F(1, 90) = 12.68, p < .001”

  • ConsistentF(1, 90) = 15.37reported p < .001 · recomputed p < .001

    “F(1, 90) = 15.37,p < .001”

  • ConsistentF(1, 90) = 18.92reported p < .001 · recomputed p < .001

    “F(1, 90) = 18.92, p < .001”

5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Suratmi and Sopandi study population conflicts with cited title

    Minor

    The text describes the cited study as an LLM-based program involving Nigerian secondary school students, while its listed title concerns teachers training elementary school students.

  • Pradana study is described as mathematics research but cited as science-skills research

    Minor

    The text attributes a focus on probability in mathematics education to Pradana and Suprapto (2020). The listed 2020 Pradana reference instead identifies a study of science-process-skills-based learning.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
de la Puente, M., Torres, J., Troncoso, A. L. B., Meza, Y. Y. H., & Carrascal, J. X. M.
Affiliations
  • Universidad de Sucre (CO)
  • Corporación Universitaria del Caribe (CO)
ORCID
Venue
Smart Learning Environments
Publisher
Springer Science and Business Media LLC
Published
2024-12-02
Volume
11
Issue
1
ISSN
2196-7091 (ISSN-L)
DOI
10.1186/s40561-024-00347-0
OpenAlex
W4404894786
Semantic Scholar
93690b99f67bf5894ca05a6bc541d4d5083c9980
Cited by
46
References
25
Citations by year
2026: 32 · 2025: 14
FWCI
17.33
Type
journal article
Language
en
Open access
Licence
Locations
  • Smart Learning Environments (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Estrategia y Datos S.A.S (2036-65-2248)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Education › Education and Critical Thinking Development
Keywords
ChatGPT, large language models, critical thinking, argumentation skills, international relations education, debate education, undergraduate students, resource-limited settings, artificial intelligence in education, developing countries

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review