The DOI matches its Crossref record; 63 of the 64 references it cites were found in a registry or an index; none of the 35 sources its claims cite is checked for support yet.
A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education
Open accessAbstract
The development of artificial intelligence in recent years has promoted the use of chatbot technology in sustainable education. Many studies examined the effect of chatbots on learning outcomes. However, scant studies summarized the effectiveness of chatbots in education. The aim of the study is to investigate the effect of chatbot-assisted learning on various components and how different moderator variables influenced the effectiveness. This study, through a meta-analysis, reviewed 32 empirical studies with 2201 participants published between 2010 and 2022. The results showed that chatbot technology exerted a medium-to-high effect on overall learning outcomes regardless of moderator variables, i.e., intervention duration, chatbot roles, and learning content. In terms of learning components, chatbots could significantly improve explicit reasoning, learning achievement, knowledge retention, and learning interest despite negative findings in critical thinking, learning engagement, and motivation. Future research could expand chatbot research by including different learning components.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This meta-analysis and systematic review examined 32 empirical studies with 2201 participants and found that chatbot technology had a medium-to-large positive overall effect on learning outcomes (d = 0.789). Effects were significant for explicit reasoning, learning achievement, knowledge retention, and learning interest, but not for critical thinking, learning engagement, or motivation.
Key findings
- The overall effect size of chatbot use in education was d = 0.789 (95% CI [0.593, 0.985], p < 0.001), described as medium-to-large.
- Chatbots significantly improved explicit reasoning (d = 1.190, p < 0.001), learning achievement (d = 1.033, p < 0.001), knowledge retention (d = 0.691, p = 0.022), and learning interest (d = 0.842, p = 0.041).
- Chatbots did not significantly improve critical thinking (d = 0.690, p = 0.144), learning engagement (d = 0.147, p = 0.114), or learning motivation (d = 0.409, p = 0.114).
- Moderator analyses found no statistically significant differences by intervention duration, chatbot roles, or learning content (all p > 0.05), though subgroup effect sizes varied.
Methods
- The authors conducted a PRISMA-guided meta-analysis and systematic review, searching major databases including Web of Science, Wiley Online Library, Springer Link, Taylor & Francis Online, Elsevier ScienceDirect, and Google Scholar on 3 September 2022.
- They included 32 empirical studies published between 2010 and 2022 with 2201 participants, calculating 76 effect sizes using Cohen’s d; studies needed experimental/control comparisons and sufficient statistics.
- Analyses used Stata MP/14.0, with publication-bias checks (funnel plot, Begg’s test, Egger’s test), heterogeneity tests (I²), sensitivity analysis, and meta-regression for intervention duration, chatbot roles, and learning content.
Limitations
- The paper states that only studies written in English were included, that only three moderator variables were examined, and that unpublished studies with insignificant results may still influence the findings despite no detected publication bias.
- Evident from the text: several outcome domains had high heterogeneity (e.g., critical thinking I² = 92.8%, motivation I² = 93.2%, overall I² = 87.8%), and some categories had very few studies, limiting moderator conclusions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
34 Not checked yet · 18 Own result · 10 No basis given · 8 Own interpretation · 5 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The meta-regression results revealed no statistically significant differences between the abovementioned values.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No significant differences in learning engagement (d = 0.147, 95% CI [-0.068, 0.363], p = 0.114) were found between chatbot-assisted and traditional learning methods.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, there were no significant differences between effect sizes of the different intervention durations (p > 0.05).”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 4 shows a significant difference in explicit reasoning (d = 1.190, 95% CI [0.868, 1.512], p < 0.001) between the experimental and control groups.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The effect size did not show a significant difference in critical thinking (d = 0.690, 95% CI [-0.235, 1.615], p = 0.144) between the chatbot-assisted and conventional learning methods.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and retention (d = 0.691, 95% CI [0.101, 1.281], p = 0.022) in the experimental group significantly improved compared with the control group.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the experimental group maintained significantly higher levels of learning interest (d = 0.842, 95% CI [0.034, 1.650], p = 0.041).”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The overall effect size of using chatbots in education was 0.789 (p < 0.001), with a 95% confidence interval between 0.593 and 0.985 (Table 4).”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, learning motivation in both groups was not significantly different (d = 0.409, 95% CI [-0.099, 0.916], p = 0.114).”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, the heterogeneity test showed that the overall effect size (Q = 582.09,I 2 = 87.8%,p < 0.001) was of considerable heterogeneity”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 20
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, Begg's test showed a presence of publication bias in learning interest (z = 2.04, p = 0.042), which was different from the result of Egger's test (t = 15.30,p = 0.056).”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.1. Analyses of Publication Bias
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The researchers calculated 76 effect sizes with a total sample size of 2201 in 32 identified studies.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 3.4. Data Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It indicates that none of the included studies could influence the pooled effect size when a specific study is omitted, confirming that the meta-analytical results are robust.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.2. Results of Sensitivity Analysis
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“They strictly followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines [54].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 3. Materials and Methods
What the sources say
- Not checked yetcites [54] Page, 2021, The PRISMA 2020 statement: An updated guideline for reporting systematic reviews, Br. Med. J. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The overall effect size in this study was thus considered as a mediumto-large level, indicating that the use of chatbot technology exerted a positive effect on learning outcomes.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“one display of explicit reasoning could associate with another one, which was the core of transactivity theory [70].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 5. Discussion
What the sources say
- Not checked yetcites [70] Pinkwart, N., and McLaren, B. (2012). Educational Technologies for Teaching Argumentation Skills, Bentham Science Publishers. Rosé, C.P. source
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Chatbots also increased college students' interest in English vocabulary learning [47].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > H6.
What the sources say
- Not checked yetcites [47] Kim, 2018, Chatbots and Korean EFL students’ English vocabulary learning, J. Digit. Converg.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Methodologically, this study included major databases and conducted a meta-analysis under the PRISMA guidelines to examine the effectiveness of chatbot technology on educational outcomes.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 6.1. Major Contributions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Chatbots could improve Chinese students' thinking ability and facilitate interactive learning [13].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 1. Introduction
What the sources say
- Not checked yetcites [13] Li, 2022, Al intelligence chatbot to improve students learning in the higher education platform, J. Interconnect. Netw. source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Chatbots also asked for clear and convincing statements, motivating students to find strong evidence.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 5. Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The LINE Bot used in military science could provide procedures for engine fan module decomposition, which improved trainee performance and reduced training costs [30].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.7. Learning Content
What the sources say
- Not checked yetcites [30] (2022, September 08). Development of Mobile Interactive Courses Based on an Artificial Intelligence Chatbot on the Communication Software LINE. Available online: https://doi.org/10.1080/10494820.2021.1937230. source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“The AsasaraBot, acting as a tutor, provided questions, encouragement, and interactions, aiming to support students' language learning [15].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.6. Chatbot Roles
What the sources say
- Not checked yetcites [15] Mageira, K., Pittou, D., Papasalouros, A., Kotis, K., Zangogianni, P., and Daradoumis, A. (2022). Educational AI chatbots for content and language integrated learning. Appl. Sci.-Basel, 12. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“A narrative-based learning system equipped with chatbot feedback significantly improved users' engagement [37].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.4. Learners' Engagement, Motivation, and Interest
What the sources say
- Not checked yetcites [37] Ruan, S., He, J.Y., Ying, R., Burkle, J., Hakim, D., Wang, A., Yin, Y.F., Zhou, L., Xu, Q.Y., and AbuHashem, A. (2020, January 21–24). Supporting Children’s Math Learning with Feedback-Augmented Narrative Technology. Proceedings of the ACM Interaction Design and Children (IDC) Conference, London, UK. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“University students created conversations with the chatbot Elbot ranging from school life to movies.After eight weeks, the experimental group better acquired vocabulary knowledge than the control group [47].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.6. Chatbot Roles
What the sources say
- Not checked yetcites [47] Kim, 2018, Chatbots and Korean EFL students’ English vocabulary learning, J. Digit. Converg.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“For example, Chen et al. [50] examined the effects of mobile devices on language learning across different intervention durations.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.5. Intervention Duration
What the sources say
- Not checked yetcites [50] Chen, 2020, The effects of using mobile devices on language learning: A meta-analysis, Educ. Technol. Res. Dev. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Chatbots improved Korean students' interest in foreign language learning [43].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > H6.
What the sources say
- Not checked yetcites [43] Han, 2020, The effects of voice-based AI chatbots on Korean EFL middle school students’ speaking competence and affective domains, Asia-Pac. J. Converg. Res. Interchange source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Given the benefits and concerns of educational chatbot use, many studies have measured the effectiveness of chatbots but have obtained inconsistent findings.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 1. Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“In nursing education, students used chatbot systems to learn courses about the physical examination, effectively enhancing students' academic performance [53].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.7. Learning Content
What the sources say
- Not checked yetcites [53] Chang, 2022, Chatbot-facilitated nursing education: Incorporating a knowledge-based chatbot system into a nursing training program, Educ. Technol. Soc.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The dynamic assessment of chatbots was effective in enhancing vocabulary retention and providing detailed information about personal learning [2].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.3. Learning Achievement and Knowledge Retention
What the sources say
- Not checked yetcites [2] (2022, September 08). Chatbot-Assisted Dynamic Assessment (CA-DA) for L2 Vocabulary Learning and Diagnosis. Available online: https://doi.org/10.1080/09588221.2021.1987272. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Chatbot interventions could effectively stimulate group discussions and help students utter their thoughts [55].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 5. Discussion
What the sources say
- Not checked yetcites [55] Tegos, 2015, Promoting academically productive talk with conversational agent interventions in collaborative learning settings, Comput. Educ. source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Moreover, explicit reasoning could increase collaboration practices, improve learning outcomes, and promote conceptual knowledge acquisition [27].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.2. Explicit Reasoning
What the sources say
- Not checked yetcites [27] Papadopoulos, 2013, ‘Make it explicit!’: Improving collaboration through increase of script coercion, J. Comput. Assist. Learn. source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“The meta-analysis of robot-assisted language learning [52] examined the influence of robot roles on the effectiveness of social robots.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.6. Chatbot Roles
What the sources say
- Not checked yetcites [52] Lee, 2022, The effects of robot-assisted language learning: A meta-analysis, Educ. Res. Rev. source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The artificial intelligence-integrated chatbot was proved effective to enhance students' thinking ability and expectations [13].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.1. Critical Thinking
What the sources say
- Not checked yetcites [13] Li, 2022, Al intelligence chatbot to improve students learning in the higher education platform, J. Interconnect. Netw. source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Fifteen instructional sessions carried out over three weeks facilitated students' comprehension of English adjectival constructions but failed to help learners generate prepositional constructions [48].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 2.5. Intervention Duration
What the sources say
- Not checked yetcites [48] Kim, R. (2022). Effects of AI Chatbot-Based Instruction on the Learning of English Adjectival Transitive Resultative Construction by Korean High School Students. [Ph.D. Thesis, Seoul National University]. Available online: https://hdl.handle.net/10371/181242.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Similarly, nursing college students using chatbots experienced a higher level of learning interest than their peers in the control group [46].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > H6.
What the sources say
- Not checked yetcites [46] Han, J.W., Park, J., and Lee, H.N. (2022). Analysis of the effect of an artificial intelligence chatbot educational program on non-face-to-face classes: A quasi-experimental study. BMC Med. Educ., 22. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The novelty effect may appear when students are introduced to new technology.The improvement of learning outcomes might result from students' newness to chatbot technology [8].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 1. Introduction
What the sources say
- Not checked yetcites [8] Fryer, 2017, Stimulating and sustaining interest in a language course: An experimental comparison of Chatbot and Human task partners, Comput. Hum. Behav. source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, the conversational behaviors in the chatbot group were considered more transactive, i.e., with more explicit arguments, than those in the control group.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 5. Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Review activities before presenting new information activate students' prior knowledge, facilitating the integration of the new and old knowledge [57].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 5. Discussion
What the sources say
- Not checked yetcites [57] Lee, 2022, Impacts of an AI-based chatbot on college students’ after-class review, academic performance, self-efficacy, learning attitude, and motivation, Educ. Technol. Res. Dev. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Since the control group in the included studies did not obtain access to chatbots, it was difficult to compare users' attitudes between the control and intervention groups.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 6.3. Implications for Future Research
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Specifically, pressure was a significant predictor of motivation in the chatbot-assisted learning context, whereas perceived competency was an influencing factor in the traditional context [18].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 5. Discussion
What the sources say
- Not checked yetcites [18] Yin, 2021, Conversation technology with micro-learning: The impact of chatbot-based learning on students’ learning motivation and performance, J. Educ. Comput. Res. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“First, the researchers only included studies written in English.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 6.2. Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, this study only included three moderator variables that did not significantly influence the effectiveness of chatbots.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 6.2. Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, the results may still be influenced by unpublished studies with insignificant results despite the absence of publication bias.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 6.2. Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Computer science and language were the most targeted fields in chatbot-based education, while engineering and mathematics received less attention [12].”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 5. Discussion
What the sources say
- Not checked yetcites [12] (2022, September 08). Interacting with Educational Chatbots: A Systematic Review. Available online: https://doi.org/10.1007/s10639-022-11177-3. source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
The included studies had sufficient statistical power to detect effects on learning engagement and motivation.
“However, significant differences in learning engagement and motivation were not found between the chatbot-based condition and the control condition.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, the study also highlights the constraints of ChatGPT, particularly its potential to limit creativity and critical thinking, which aligns with the findings of Deng and Yu (2023) but contrasts with Abramson's (2023) view that AI can enhance creativity by offloading routine tasks.”
From Motivations and Affordances of ChatGPT Usage for College Students’ Learning (2025) · cites this paper as Deng and Yu (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This notable increase may be associated with interactive chatbot engagement; according to Deng and Yu (2023), one-way and two-way text message exchanges retain approximately 25% more knowledge.”
From SiKurang: Development and Field Evaluation of an Offline-First mHealth System for Community-Based Stunting Risk Detection and Counselling in Low-Connectivity Primary Care Settings (2026) · cites this paper as Deng and Yu (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“according to Deng and Yu (2023), one-way and two-way text message exchanges retain approximately 25% more knowledge.”
From SiKurang: Development and Field Evaluation of an Offline-First mHealth System for Community-Based Stunting Risk Detection and Counselling in Low-Connectivity Primary Care Settings (2026) · cites this paper as Deng and Yu (2023)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This aligns with a meta-analysis conducted by Deng and Yu [31], which indicated that chatbot programs can serve three roles in education—teaching assistant, personal tutor, and learning partner (Fig. 2).”
From Development and effects of a chatbot education program for self-directed learning in nursing students (2025) · cites this paper as [31]
In the citing paper
“AI tools, such as chatbots, have been shown to enhance student interaction and improve learning outcomes (Deng & Yu, 2023;Rasul et al., 2023).”
From Ethical use of artificial intelligence based tools in higher education: Are future business leaders ready? (2025) · cites this paper as Deng & Yu, 2023
In the citing paper
“chatbots, including ChatGPT, have demonstrated the potential to reshape and revolutionize education, creating new, novel opportunities for enhancing the learning process and outcomes [83], facilitating different learning approaches, and offering a range of pedagogical benefits [19, 43, 72]”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [19]
In the citing paper
“chatbots, including ChatGPT, have the potential to significantly impact students' thinking abilities, including retention, reasoning, analysis skills [19, 45], and foster innovation and creativity capabilities [83]”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [19]
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = 1.47reported p = .142 · recomputed p = .142
“z = 1.47, p = .142”
- Consistentz = 1.32reported p = .188 · recomputed p = .187
“z = 1.32, p = .188”
- Consistentz = 0.53reported p = .598 · recomputed p = .596
“z = 0.53, p = .598”
- Consistentz = 1.35reported p = .176 · recomputed p = .177
“z = 1.35, p = .176”
- Consistentz = 1.23reported p = .217 · recomputed p = .219
“z = 1.23, p = .217”
- Consistentz = 2.04reported p = .042 · recomputed p = .041
“(z = 2.04, p = .042”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| 8 | H. L. Chen et al. (2020) [17] | 58 (19/29) | 4 weeks | Tutor | Language | Learning achievement and retention |”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 6.3. Implications for Future Research
Reported 58 · recomputed 48
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 4.3. Results of Moderator Analyses
“No significant differences in learning engagement (d = 0.147, 95% CI [-0.068, 0.363], p = 0.114) were found between chatbot-assisted and traditional learning methods.”
§ A Meta-Analysis and Systematic Review of the Effect of Chatbot Technology Use in Sustainable Education > 6.3. Implications for Future Research
“| 5 | Learning engagement | 0.147 | -0.068, 0.363 | 1.34 | 0.180 | 3.22 | 6 | 0.780 | 0.0% |”
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139 · cited 1×
“The heterogeneity will be considered low if I2 is less than 25%, moderate if I2 falls between 25% and 75%, and substantial if I2 is greater than 75% [68].”
BackgroundIntrinsic and Extrinsic Motivations: Classic Definitions and New Directions
Ryan, R. M., & Deci, E. L. (2000) · Contemporary Educational Psychology · doi:10.1006/ceps.1999.1020 · Cited by 18,666 · cited 1×
“Intrinsic motivation focuses on learners’ inner satisfaction, whereas extrinsic motivation is defined as the behavior for external and separable results [42].”
BackgroundToward a hierarchical model of intrinsic and extrinsic motivation
Vallerand, R. J. (1997) · Advances in experimental social psychology · doi:10.1016/s0065-2601(08)60019-2 · Cited by 3,198 · cited 1×
“Motivation can be categorized into intrinsic and extrinsic motivations [41].”
BackgroundChatbots applications in education: A systematic review
Okonkwo, C. W., & Ade-Ibijola, A. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100033 · Cited by 879 · cited 3×
“[1], Okonkwo, and Ade-Ibijola [6], Perez et al.”
BackgroundInteracting with educational chatbots: A systematic review
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11177-3 · Cited by 841 · cited 3×
“Computer science and language were the most targeted fields in chatbot-based education, while engineering and mathematics received less attention [12].”
BackgroundHuang, W., Hew, K. F., & Fryer, L. K. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12610 · Cited by 830
Interest, motivation and learning: An educational-psychological perspective
Krapp, A. (1999) · European Journal of Psychology of Education · doi:10.1007/bf03173109 · Cited by 595 · cited 1×
“Interest indicates the individual readiness or predisposition to engage in a given learning task with effort [45].”
BackgroundA review of opportunities and challenges of chatbots in education
Hwang, G.-J., & Chang, C.-Y. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1952615 · Cited by 530
Rediscovering the use of chatbots in education: A systematic literature review
Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22326 · Cited by 483 · cited 1×
“Problems have also included technological limitations and training side effects [7].”
Background
Research works printed but not linked: 6
- Lee, 2022, A meta-analysis of the effects of using AI chatbot in Korean EFL education, Stud. Engl. Lang. Lit., 48, 213
- Sternberg, R.J., and Preiss, D.D. (2010). Innovations in Educational Psychology: Perspectives on Learning, Teaching and Human Development, Springer.
- Cole, P.G., and Chan, L.K.S. (1994). Teaching Principles and Practice, Prentice Hall.
- Kim, R. (2022). Effects of AI Chatbot-Based Instruction on the Learning of English Adjectival Transitive Resultative Construction by Korean High School Students. [Ph.D. Thesis, Seoul National University]. Available online: https://hdl.handle.net/10371/181242.
- Abbasi, 2019, Effect of chatbot systems on student’s learning outcomes, Sylwan, 163, 49
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, Routledge. [2nd ed.].
Cited by
Sorted by citations
Role of AI chatbots in education: systematic literature review
Labadze, L., Grigolia, M., & Machaidze, L. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00426-1 · Cited by 1,043
“Deng and Yu (2023) found that chatbots had a significant and positive influence on numerous learning-related aspects but they do not significantly improve motivation among students.”
BackgroundFan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13544 · Cited by 805
The role of ChatGPT in higher education: Benefits, challenges, and future research directions
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.29 · Cited by 561
“Nonetheless, adaptive learning systems aim to transform students from passive recipients to active collaborators through a scaffolded approach to learning (Deng & Yu, 2023; Zawacki-Richter et al., 2019).”
BackgroundWhat ChatGPT means for universities: Perceptions of scholars and students
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.22 · Cited by 431
“…technologies can enhance student interaction and learning processes (D’Mello et al., 2014), enrich learning experiences by impacting student success in higher education (Winkler & Söllner, 2018), and potentially improve student motivation, engagement, and learning outcomes (Deng & Yu, 2023).”
BackgroundChang, D. H., Lin, M. P.-C., Hajian, S., & Wang, Q. Q. (2023) · Sustainability · doi:10.3390/su151712921 · Cited by 370
“In addition, according to a meta-analysis conducted by Deng and Yu [38], chatbots can potentially have a medium-to-high effect on achievement or learning outcomes.”
Method
Metadata
- Authors
- Deng, X., & Yu, Z.
- Affiliations
- Beijing Language and Culture University (CN)
- Corresponding author
- Yu, Zhonggen
- Venue
- Sustainability
- Publisher
- MDPI AG
- Published
- 2023-02-06
- Volume
- 15
- Issue
- 4
- Pages
- 2940
- ISSN
- 2071-1050 (ISSN-L)
- OpenAlex
- W4319440682
- Semantic Scholar
- 27f4d355b27a6878332a87845a6ff13940fe9158
- Cited by
- 298
- References
- 64
- Citations by year
- 2026: 76 · 2025: 115 · 2024: 72 · 2023: 35
- FWCI
- 12.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Sustainability (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- 2019 MOOC of Beijing Language and Culture University (MOOC201902) (important) “Introduction to Linguistics” (202010032003, SJTS202108, 22YCX017)
- “Introduction to Linguistics” of online and offline mixed courses in Beijing Language and Culture University in 2020 (202010032003, SJTS202108, 22YCX017)
- Beijing Co-construction Project Research and reform of the “Undergraduate Teaching Reform and Innovation Project” of Beijing higher education in 2020-innovative “multilingual +” excellent talent training system (202010032003, SJTS202108, 22YCX017)
- The research project of Graduate Students of Beijing Language and Culture University “Xi Jinping: The Governance of China” (202010032003, SJTS202108, 22YCX017)
- Beijing Language and Culture University (202010032003, SJTS202108, 22YCX017)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- chatbots, chatbot-assisted learning, sustainable education, learning outcomes, moderators, explicit reasoning, knowledge retention, academic performance, student learning interest, critical thinking, student engagement, student motivation, artificial intelligence in education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
