Cited by 576Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningAI in Service Interactions

Do AI chatbots improve students learning outcomes? Evidence from a meta‐analysis

Open access
Wu, Rong, Yu, Zhonggen·2024·British Journal of Educational Technology

Abstract

Artificial intelligence (AI) chatbots are gaining increasing popularity in education. Due to their increasing popularity, many empirical studies have been devoted to exploring the effects of AI chatbots on students' learning outcomes. The proliferation of experimental studies has highlighted the need to summarize and synthesize the inconsistent findings about the effects of AI chatbots on students' learning outcomes. However, few reviews focused on the meta‐analysis of the effects of AI chatbots on students' learning outcomes. The present study performed a meta‐analysis of 24 randomized studies utilizing Stata software (version 14). The main goal of the current study was to meta‐analytically examine the effects of AI chatbots on students' learning outcomes and the moderating effects of educational levels and intervention duration. The results indicated that AI chatbots had a large effect on students' learning outcomes. Moreover, AI chatbots had a greater effect on students in higher education, compared to those in primary education and secondary education. In addition, short interventions were found to have a stronger effect on students' learning outcomes than long interventions. It could be explained by the argument that the novelty effects of AI chatbots could improve learning outcomes in short interventions, but it has worn off in the long interventions. Future designers and educators should make attempt to increase students' learning outcomes by equipping AI chatbots with human‐like avatars, gamification elements and emotional intelligence. Practitioner notes: What is already known about this topic: In recent years, artificial intelligence (AI) chatbots have been gaining increasing popularity in education. Studies undertaken so far have provided conflicting evidence concerning the effects of AI chatbots on students' learning outcomes. There has remained a paucity of meta‐analyses synthesizing the contradictory findings about the effects of AI chatbots on students' learning outcomes. What this paper adds: This study, through meta‐analysis, synthesized these recent findings about the effects of AI chatbots on students' learning outcomes. This study found that AI chatbots could have a large effect on students' learning outcomes. This study found that the effects of AI chatbots were moderated by educational levels and intervention duration. Implications for practice and/or policy: AI chatbot designers could make AI chatbots better by equipping AI chatbots with human‐like avatars, gamification elements and emotional intelligence Practitioners and/or teachers should draw attention to the positive and negative effects of AI chatbots on students. Considering the importance of ChatGPT, more research is required to develop a better understanding of the effects of ChatGPT in education. More research is needed to examine the mechanisms underlying the effects of AI chatbots on students' learning outcomes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/bjet.13334

Cites

78 of 78 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is a dearth of RCTs conducted in authentic K-12 school settings, with most studies taking place in higher education contexts.

    “There is also a dearth of RCTs conducted in authentic K-12 school settings, with most studies taking place in higher education contexts (Wu & Yu, 2024 ).”

    From Enhancing School Students' Self-Regulated Learning through Generative AI Support: A Randomized Controlled Trial (2026) · cites this paper as Wu & Yu, 2024

    The Additional information table (page unavailable) lists 19 of 24 included studies in higher education and five in primary or secondary education, supporting the claim that most were in higher education. The eligibility criteria allow both randomized controlled and quasi-experimental designs, while the Limitations section says learning locations could not be examined. The paper therefore does not establish a dearth specifically of RCTs in authentic K–12 school settings. (LLM, conf=0.87)

    Experimental — accuracy not yet measured
  • Interventions lasting less than 1 week generally failed to statistically significantly affect student learning or motivation.

    “Wu and Yu ( 2024) found that interventions lasting less than 1 week generally failed to statistically significantly affect student learning or motivation.”

    From Enhancing School Students' Self-Regulated Learning through Generative AI Support: A Randomized Controlled Trial (2026) · cites this paper as Wu and Yu (2024)

    The Study coding section divides interventions at 10 weeks, and the results report a significant effect for the under-10-week group. The paper does not report a separate analysis of interventions lasting less than one week, so that result cannot establish whether such interventions generally failed to significantly affect learning or motivation. I searched for a one-week subgroup and its significance results. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. [30]30. Kim, H. S., Kim, N. Y., & Cha, Y. J. (2021). Effects of AI chatbots on EFL students' communication skills. Korean Journal of English Language and Linguistics, 21, 712-734. https://doi.org/10.15738/ KJELL.21.202108.712

Cited by

Sorted by citations

Authors
Wu, R., & Yu, Z.
Affiliations
Beijing Language and Culture University (CN)
Corresponding author
Yu, Zhonggen
Venue
British Journal of Educational Technology
Publisher
Wiley
Published
2023-05-03
Volume
55
Issue
1
Pages
10-33
ISSN
  • 0007-1013 (ISSN-L)
  • 1467-8535
OpenAlex
W4367849331
Cited by
576
References
76
Citations by year
2026: 198 · 2025: 245 · 2024: 113 · 2023: 20
FWCI
134.94
Type
journal article
Language
en
Open access
Locations
British Journal of Educational Technology (publishedVersion)
Funders
  • Beijing Language and Culture University (202010032003, MOOC201902)
  • Fundamental Research Funds for the Central Universities (MOOC201902, 202010032003)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
Keywords
AI chatbots, learning outcomes, educational technology, higher education, primary education, secondary education, intervention duration, novelty effect, gamification, emotional intelligence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review