Does Chatting with Chatbots Improve Language Learning Performance? A Meta-Analysis of Chatbot-Assisted Language Learning

Wang, Feifei; Cheung, Alan C. K.; Neitzel, Amanda J.; Chai, Ching Sing

doi:10.3102/00346543241255621

Cited by 64EFL/ESL Teaching and LearningEnglish Language Learning and TeachingIntelligent Tutoring Systems and Adaptive Learning

Does Chatting with Chatbots Improve Language Learning Performance? A Meta-Analysis of Chatbot-Assisted Language Learning

Wang, Feifei, Cheung, Alan C. K., Neitzel, Amanda J., Chai, Ching Sing·2025·Review of Educational Research

DOI 10.3102/00346543241255621

Open this paper in CoChat — free account

Abstract

Given the importance of conversation practice in language learning, chatbots, especially ChatGPT, have attracted considerable attention for their ability to converse with learners using natural language. This review contributes to the literature by examining the currently unclear overall effect of using chatbots on language learning performance and comprehensively identifying important study characteristics that affect the overall effectiveness. We meta-analyzed 70 effect sizes from 28 studies, using robust variance estimation. The effects were assessed based on 18 study characteristics about learners, chatbots, learning objectives, context, communication/interaction, and methodological and pedagogical designs. Results indicated that using chatbots produced a positive overall effect on language learning performance ( g = 0.484), compared to nonchatbot conditions. Moreover, four characteristics (i.e., educational level, language level, interface design, and interaction capability) affected the overall effectiveness. In an in-depth discussion on how the 18 characteristics are related to the effectiveness, future implications for practice and research are presented.

Integrity signals

Registry

Crossref · 10.3102/00346543241255621

Cites

92 of 92 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The prevalence of Conversational and NLP Agents reflects sustained interest in dialogue-based learning tools for language acquisition and tutoring.
“The prevalence of Conversational and NLP Agents reflects sustained interest in dialogue-based learning tools for language acquisition and tutoring, consistent with prior findings on chatbot-mediated instruction [29].”
From Systematic review of artificial intelligence in education: trends, benefits, and challenges (2025) · cites this paper as [29]

Untitled work
(1978) · Medical Entomology and Zoology · Cited by 53,488 · see citing works
Conducting Meta-Analyses in <i>R</i> with the <b>metafor</b> Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464 · see citing works
Introduction to Meta‐Analysis
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225 · see citing works
Untitled work
(2000) · Cited by 9,198 · see citing works
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · see citing works
On seeing human: A three-factor theory of anthropomorphism
Epley, N., Waytz, A., & Cacioppo, J. T. (2007) · Psychological Review · doi:10.1037/0033-295x.114.4.864 · Cited by 4,069 · cited 1× · see citing works
“The third feature, chatbot anthropomorphism, means that chatbots use verbal cues, such as thanking, self-disclosure, or excuse, to interact with learners, showing humanlike characteristics, behaviors, and emotions (Adam et al., 2021; Epley et al., 2007).”
Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., & Moher, D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.003 · Cited by 3,490 · see citing works
What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00237-x · Cited by 2,003 · cited 1× · see citing works
“Among the discussions, there has been an ongoing debate on whether chatbots are devils or angels (Tlili et al., 2023), especially in the field of language education due to chatbots’ autonomous capacity to converse with learners in the target language as tireless practice partners and information…”
ChatGPT for Language Teaching and Learning
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023) · RELC Journal · doi:10.1177/00336882231162868 · Cited by 1,371 · cited 1× · see citing works
“Some researchers anticipate that the chatbots will bring in a significant shift in language learning as powerful tools (e.g., Kohnke et al., 2023), and others express concerns about these tools for generating wrong information, potential substitution of human teachers, and ethical considerations…”
AI-based chatbots in customer service and their effects on user compliance
Adam, M., Wessel, M., & Benlian, A. (2021) · Electronic Markets · doi:10.1007/s12525-020-00414-7 · Cited by 1,195 · cited 1× · see citing works
“The third feature, chatbot anthropomorphism, means that chatbots use verbal cues, such as thanking, self-disclosure, or excuse, to interact with learners, showing humanlike characteristics, behaviors, and emotions (Adam et al., 2021; Epley et al., 2007).”
Professional Development in Integrating Technology Into Teaching and Learning: Knowns, Unknowns, and Ways to Pursue Better Questions and Answers
Lawless, K. A., & Pellegrino, J. W. (2007) · Review of Educational Research · doi:10.3102/0034654307309921 · Cited by 1,172 · see citing works
International Journal of Scientific and Technology Research
Kebede, A., & Woldewahid, G. (2014) · Cited by 1,049 · see citing works
Interacting with educational chatbots: A systematic review
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11177-3 · Cited by 841 · see citing works
Chatbots for language learning—Are they really useful? A systematic review of chatbot‐supported language learning
Huang, W., Hew, K. F., & Fryer, L. K. (2022) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12610 · Cited by 830 · see citing works
The persona effect
Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, S., Stone, B. A., & Bhogal, R. S. (1997) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/258549.258797 · Cited by 812 · see citing works
Small sample adjustments for robust variance estimation with meta-regression.
Tipton, E. (2015) · Psychological Methods · doi:10.1037/met0000011 · Cited by 810 · see citing works
Chatbots for learning: A review of educational chatbots for the Facebook Messenger
Smutny, P., & Schreiberova, P. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103862 · Cited by 770 · see citing works
The power of statistical tests in meta-analysis
Hedges, L. V., & Pigott, T. D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.203 · Cited by 716 · cited 2× · see citing works
“Because the methods for calculating the power do not account for multiple effect sizes in each study (Hedges & Pigott, 2001), this is only an approximation and must be interpreted with caution.”
Large language models in education: a focus on the complementary relationship between human teachers and ChatGPT
Jeon, J., & Lee, S. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11834-1 · Cited by 696 · see citing works
Methodological guidance paper: High-quality meta- analysis in a systematic review
Pigott, T. D., & Polanin, J. R. (2020) · Review of Educational Research · doi:10.3102/0034654319877153 · Cited by 568 · see citing works
A review of opportunities and challenges of chatbots in education
Hwang, G.-J., & Chang, C.-Y. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1952615 · Cited by 530 · see citing works
How Methodological Features Affect Effect Sizes in Education
Cheung, A. C. K., & Slavin, R. E. (2016) · Educational Researcher · doi:10.3102/0013189x16656615 · Cited by 529 · see citing works
Self-citation
How meta-analysis increases statistical power.
Cohn, L. D., & Becker, B. J. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.3.243 · Cited by 500 · see citing works
In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias
Terrin, N. C., Schmid, C. H., & Lau, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.006 · Cited by 493 · see citing works
Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes.
Rodgers, M. A., & Pustejovsky, J. E. (2021) · Psychological Methods · doi:10.1037/met0000300 · Cited by 492 · cited 1× · see citing works
“Other methods to test for and assess the degree of publication bias exist, but few have been utilized with robust variance estimation and may rely on larger sample sizes (Rodgers & Pustejovsky, 2021).”
Chatbot learning partners: connecting learning experiences, interest and competence
Fryer, L. K., Nakao, K., & Thompson, A. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.12.023 · Cited by 470 · see citing works
A Theory of Learning for the Mobile Age
Sharpies, M., Taylor, J., & Vavoula, G. (2007) · SAGE Publications Ltd eBooks · doi:10.4135/9781848607859.n10 · Cited by 454 · see citing works
Stimulating and sustaining interest in a language course: An experimental comparison of Chatbot and Human task partners
Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., & Sherlock, Z. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.05.045 · Cited by 448 · see citing works
A systematic review of conversational AI in language education: focusing on the collaboration with human teachers
Ji, H., Han, I., & Ko, Y. (2023) · Journal of Research on Technology in Education · doi:10.1080/15391523.2022.2142873 · Cited by 394 · see citing works
Roles and research foci of artificial intelligence in language education: an integrated bibliographic analysis and systematic review approach
Liang, J.-C., Hwang, G.-J., Chen, M.-R. A., & Darmawansah, D. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1958348 · Cited by 378 · see citing works
Educational AI Chatbots for Content and Language Integrated Learning
Mageira, K., Pittou, D., Papasalouros, A., Kotis, K., Zangogianni, P., & Daradoumis, A. (2022) · Applied Sciences · doi:10.3390/app12073239 · Cited by 372 · see citing works
A general linear model for estimating effect size in the presence of publication bias
Vevea, J. L., & Hedges, L. (1995) · Psychometrika · doi:10.1007/bf02294384 · Cited by 367 · see citing works
Implementation of an AI chatbot as an English conversation partner in EFL speaking classes
Yang, H., Kim, H., Lee, J. H., & Shin, D. (2022) · ReCALL · doi:10.1017/s0958344022000039 · Cited by 334 · see citing works
Effective Reading Programs for the Elementary Grades: A Best-Evidence Synthesis
Slavin, R. E., Lake, C., Chambers, B., Cheung, A., & Davis, S. (2009) · Review of Educational Research · doi:10.3102/0034654309341374 · Cited by 319 · see citing works
Self-citation
Using Chatbots as AI Conversational Partners in Language Learning
Belda-Medina, J., & Calvo-Ferrer, J. R. (2022) · Applied Sciences · doi:10.3390/app12178427 · Cited by 306 · see citing works
A conceptual framework based on Activity Theory for mobile CSCL
Zurita, G., & Nussbaum, M. (2006) · British Journal of Educational Technology · doi:10.1111/j.1467-8535.2006.00580.x · Cited by 247 · see citing works
Exploring the Implications of ChatGPT for Language Learning in Higher Education
Baskara, R., & Mukarto, M. (2023) · IJELTAL (Indonesian Journal of English Language Teaching and Applied Linguistics) · doi:10.21093/ijeltal.v7i2.1387 · Cited by 247 · see citing works
The SAGE Handbook of E-learning Research
Andrews, R., & Haythornthwaite, C. (2007) · SAGE Publications Ltd eBooks · doi:10.4135/9781848607859 · Cited by 243 · see citing works
An analysis of children’ interaction with an AI chatbot and its impact on their interest in reading
Liu, C.-C., Liao, M.-G., Chang, C.-H., & Lin, H.-M. (2022) · Computers & Education · doi:10.1016/j.compedu.2022.104576 · Cited by 240 · see citing works
A ChatBot for Learning Chinese: Learning Achievement and Technology Acceptance
Chen, H.-L., Vicki Widarso, G., & Sutrisno, H. (2020) · Journal of Educational Computing Research · doi:10.1177/0735633120929622 · Cited by 220 · see citing works
Chatbot to improve learning punctuation in Spanish and to enhance open and flexible learning environments
Cano, E. V., Andrés, S. M., & Menéses, E. L. (2021) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-021-00269-8 · Cited by 220 · see citing works
Chatbot-assisted dynamic assessment (CA-DA) for L2 vocabulary learning and diagnosis
Jeon, J. (2023) · Computer Assisted Language Learning · doi:10.1080/09588221.2021.1987272 · Cited by 202 · see citing works
The Effects of Mobile-Computer-Supported Collaborative Learning: Meta-Analysis and Critical Synthesis
Sung, Y., Yang, J.-M., & Lee, H.-Y. (2017) · Review of Educational Research · doi:10.3102/0034654317704307 · Cited by 197 · see citing works
Multimedia-Assisted Language Learning
(2020) · Multimedia-Assisted Language Learning · doi:10.15702/mall · Cited by 193 · see citing works
Same benefits, different communication patterns: Comparing Children's reading with a conversational agent vs. a human partner
Xu, Y., Wang, D., Collins, P., Lee, H., & Warschauer, M. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.104059 · Cited by 180 · see citing works
Effective Reading Programs for Spanish-Dominant English Language Learners (ELLs) in the Elementary Grades
Cheung, A., & Slavin, R. (2012) · Review of Educational Research · doi:10.3102/0034654312465472 · Cited by 155 · see citing works
Self-citation
The effectiveness of intelligent tutoring systems on K‐12 students' reading comprehension: A meta‐analysis
Xu, Z., Wijekumar, K. K., Ramı́rez, G., Hu, X., & Irey, R. C. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12758 · Cited by 145 · see citing works
A systematic review of research on speech-recognition chatbots for language learning: Implications for future directions in the era of large language models
Jeon, J., Lee, S., & Choi, S. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2023.2204343 · Cited by 143 · see citing works
Effects of a vocabulary acquisition and assessment system on students' performance in a blended learning class for English subject
Jia, J., Chen, Y., Ding, Z., & Ruan, M. (2012) · Computers & Education · doi:10.1016/j.compedu.2011.08.002 · Cited by 142 · see citing works
Effects of the Attributes of Educational Interventions on Students’ Academic Performance
de Boer, H., Donker, A. S., & van der Werf, G. P. C. (2014) · Review of Educational Research · doi:10.3102/0034654314540006 · Cited by 141 · see citing works
Engaging high school students using chatbots
Benotti, L., Martínez, M. C., & Schapachnik, F. (2014) · doi:10.1145/2591708.2591728 · Cited by 139 · see citing works
ArgueTutor: An Adaptive Dialog-Based Learning System for Argumentation Skills
Wambsganss, T., Kueng, T., Soellner, M., & Leimeister, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445781 · Cited by 138 · see citing works
Learning L2 pronunciation with a mobile speech recognizer: French /y/
Liakin, D., Cardoso, W., & Liakina, N. (2015) · CALICO Journal · doi:10.1558/cj.v32i1.25962 · Cited by 132 · see citing works
Adding communicative and affective strategies to an embodied conversational agent to enhance second language learners' willingness to communicate
Ayedoun, E., Hayashi, Y., & Seta, K. (2019) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-018-0171-6 · Cited by 128 · see citing works
Comparing immersive Virtual Reality to mobile applications in foreign language learning in higher education: a quasi-experiment
Nicolaidou, I. A., Pissas, P., & Boglou, D. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2020.1870504 · Cited by 125 · see citing works
Acceptance of Educational Use of AI Chatbots in the Context of Self-Directed Learning with Technology and ICT Self-Efficacy of Undergraduate Students
Eşiyok, E., Gökçearslan, Ş., & Küçükergi̇n, K. G. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2303557 · Cited by 122 · see citing works
Proposing a task-oriented chatbot system for EFL learners speaking practice
Hsu, M.-H., Chen, P.-S., & Yu, C.-S. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2021.1960864 · Cited by 119 · see citing works
Dialogue with a conversational agent promotes children’s story comprehension via enhancing engagement
Xu, Y., Aubele, J., Vigil, V., Bustamante, A. S., Kim, Y.-S., & Warschauer, M. (2022) · Child Development · doi:10.1111/cdev.13708 · Cited by 113 · see citing works
A Study on the Use of Artificial Intelligence Chatbots for Improving English Grammar Skills
Kim, N. (2019) · Journal of Digital Convergence · doi:10.14400/jdc.2019.17.8.037 · Cited by 107 · see citing works
Effect of chatbot-assisted language learning: A meta-analysis
Zhang, S., Shan, C., Lee, J. S. Y., Che, S., & Kim, J. H. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11805-6 · Cited by 103 · see citing works
Conversation with a Chatbot before an Online EFL Group Discussion and the Effects on Critical Thinking
Goda, Y., Yamada, M., Matsukawa, H., Hata, K., & Yasunami, S. (2014) · The Journal of Information and Systems in Education · doi:10.12937/ejsise.13.1 · Cited by 102 · see citing works
The rise of chatbots – new personal assistants in foreign language learning
Dokukina, I. V., & Gumanova, J. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.02.212 · Cited by 99 · see citing works
Intelligent personal assistants for autonomous second language learning: An investigation of Alexa
Dizon, G., & Tang, D. (2020) · The JALT CALL Journal · doi:10.29140/jaltcall.v16n2.273 · Cited by 97 · see citing works
EnglishBot: An AI-Powered Conversational System for Second Language Learning
Ruan, S., Jiang, L., Xu, Q., Liu, Z., Davis, G. M., Brunskill, E., & Landay, J. A. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450648 · Cited by 96 · see citing works
Effects of self-regulation scaffolding on online participation and learning outcomes
Song, D., & Kim, D. (2021) · Journal of Research on Technology in Education · doi:10.1080/15391523.2020.1767525 · Cited by 95 · see citing works
The impact of intelligent personal assistants on adolescent EFL learners’ listening comprehension
Tai, T.-Y., & Chen, H. H.-J. (2024) · Computer Assisted Language Learning · doi:10.1080/09588221.2022.2040536 · Cited by 93 · see citing works
The effects of robot-assisted language learning: A meta-analysis
Lee, H., & Lee, J. H. (2021) · Educational Research Review · doi:10.1016/j.edurev.2021.100425 · Cited by 92 · see citing works
The effects of voice-based AI chatbots on Korean EFL middle school students’ speaking competence and affective domains
Han, D.-E. (2020) · Asia-pacific Journal of Convergent Research Interchange · doi:10.47116/apjcri.2020.07.07 · Cited by 90 · see citing works
Effects of intelligent personal assistants on EFL learners’ oral proficiency outside the classroom
Tai, T. (2022) · Computer Assisted Language Learning · doi:10.1080/09588221.2022.2075013 · Cited by 88 · see citing works
Investigating the impact of the Amazon Alexa on the development of L2 listening and speaking skills
Hsu, H.-L., Chen, H. H.-J., & Todd, A. G. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2021.2016864 · Cited by 84 · see citing works
Facilitating authentic contextual EFL speaking and conversation with smart mechanisms and investigating its influence on learning achievements
Hwang, W., Guo, B.-C., Hoang, A., Chang, C., & Wu, N.-T. (2022) · Computer Assisted Language Learning · doi:10.1080/09588221.2022.2095406 · Cited by 83 · see citing works
Chapter 3. From input, output and comprehension to negotiation, evidence, and attention: An overview of theory and research on learner interaction and SLA
Pica, T. (2013) · AILA Applied Linguistics Series · doi:10.1075/aals.9.06ch3 · Cited by 81 · see citing works
Can home use of speech-enabled artificial intelligence mitigate foreign language anxiety: investigation of a concept
Bao, M. (2019) · Arab World English Journal · doi:10.24093/awej/call5.3 · Cited by 70 · see citing works
A Study on Chatbots for Developing Korean College Students' English Listening and Reading Skills
Kim, H.-S. (2018) · Journal of Digital Convergence · doi:10.14400/jdc.2018.16.8.019 · Cited by 64 · see citing works
Survey of Various AI Chatbots Based on Technology Used
Singh, S., & Thakur, H. K. (2020) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito48877.2020.9197943 · Cited by 64 · see citing works
Dialogue systems for language learning: A meta-analysis
Bibauw, S., Noortgate, W. V. D., François, T., & Desmet, P. (2022) · Language learning & technology · doi:10.64152/10125/73488 · Cited by 63 · see citing works
What kind of individual support activities in interventions foster pre-service and beginning teachers’ self-efficacy? A meta-analysis
Mok, S. Y., Rupp, D., & Holzberger, D. (2023) · Educational Research Review · doi:10.1016/j.edurev.2023.100552 · Cited by 59 · see citing works
Chatbots and Korean EFL Students' English Vocabulary Learning
김나영 (2018) · Journal of Digital Convergence · doi:10.14400/jdc.2018.16.2.001 · Cited by 58 · see citing works
The impact of intelligent personal assistants on adolescent EFL learners’ speaking proficiency
Tai, T., & Chen, H. H.-J. (2022) · Computer Assisted Language Learning · doi:10.1080/09588221.2022.2070219 · Cited by 56 · see citing works
Architecture of a Framework for Generic Assisting Conversational Agents
Sansonnet, J., Leray, D., & Martin, J. (2006) · Lecture notes in computer science · doi:10.1007/11821830_12 · Cited by 49 · see citing works
Applying Ai Chatbot For Teaching A Foreign Language: An Empirical Research
Nghi, T. T., Phúc, T. H., & Thắng, N. T. (2019) · International journal of scientific and technology research · Cited by 48 · see citing works
The Role of Chatbots in Foreign Language Learning: The Present Situation and the Future Outlook
Petrović, J., & Jovanovic, M. (2021) · Studies in computational intelligence · doi:10.1007/978-3-030-72711-6_17 · Cited by 43 · see citing works
Foreign language education policy on the horizon
Hult, F. M. (2018) · Foreign Language Annals · doi:10.1111/flan.12315 · Cited by 27 · see citing works
How to Teach Pre-Service Teachers to Make a Didactic Program? The Collaborative Learning Associated with Mobile Devices
Costa, R. S., Medrano, M. M., Ostáriz, P. L., & Moreno-Guerrero, A.-J. (2020) · Sustainability · doi:10.3390/su12093755 · Cited by 22 · see citing works
Dialogue-based CALL: an overview of existing research
Bibauw, S., François, T., & Desmet, P. (2015) · EUROCALL Conference · doi:10.14705/rpnet.2015.000310 · Cited by 13 · see citing works
Investigating the impact of context-awareness smart learning mechanism on EFL conversation learning
Liu, Y., Hwang, W., & Su, C.-H. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2023.2194931 · Cited by 13 · see citing works
Escapeling: A Gamified, AI-Supported Chatbot for Collaborative Language Practice
Johnson, C., Urazov, M., & Zanoli, E. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-90677-1_14 · Cited by 12 · see citing works
Conversational recommendation: A grand AI challenge
Jannach, D., & Chen, L. (2022) · AI Magazine · doi:10.1002/aaai.12059 · Cited by 12 · see citing works
The Tools That Shape Us: Composing by Hand vs. Composing by Machine
Madigan, C. (1984) · English Education · doi:10.58680/ee198415052 · Cited by 8 · see citing works
The Effect of Oral Practice via Chatbot on Students’ Oral English Accuracy
Ye, Y., Deng, J., & Liu, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92836-0_30 · Cited by 6 · see citing works
Improving a Gamified Language Learning Chatbot Through AI and UX Boosting
Tsvilodub, P., Chevalier, E., Klütz, V., Oberbeck, T., Sigetova, K., & Wollatz, F. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-21569-8_52 · Cited by 5 · see citing works
An Analysis of High School Students’ Task Interaction Using AI Chatbots
Park, M. H., & Kim, J. R. (2021) · Korean Association For Learner-Centered Curriculum And Instruction · doi:10.22251/jlcci.2021.21.9.1 · Cited by 3 · see citing works

Flagged first, then by citations

1 of the 93 references couldn’t be matched to a known paper.

Sorted by citations

Systematic review of artificial intelligence in education: trends, benefits, and challenges
Garzón, J., Patiño, E., & Marulanda, C. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9080084 · Cited by 265 · see citing works
“The prevalence of Conversational and NLP Agents reflects sustained interest in dialogue-based learning tools for language acquisition and tutoring, consistent with prior findings on chatbot-mediated instruction [29].”
Exploring the impact of generative artificial intelligence on students’ learning outcomes: a meta-analysis
Zhu, Y., Liu, Q., & Zhao, L. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13420-z · Cited by 92 · see citing works
Chatbots in education: Hype or help? A meta-analysis
Laun, M., & Wolff, F. (2025) · Learning and Individual Differences · doi:10.1016/j.lindif.2025.102646 · Cited by 85 · see citing works
Effectiveness of Chatbots in Improving Language Learning: A Meta‐Analysis of Comparative Studies
Lyu, B., Lai, C., & Guo, J. (2024) · International Journal of Applied Linguistics · doi:10.1111/ijal.12668 · Cited by 67 · see citing works
L2 Students' Speaking Skills in Robot‐Assisted Language Learning: A Meta‐Analysis
Wang, Y., Pan, Z., & Solhi, M. (2025) · European Journal of Education · doi:10.1111/ejed.70416 · Cited by 42 · see citing works
Effects of GenAI Interventions on Student Academic Performance: A Meta-Analysis
Gu, J., & Yan, Z. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251349620 · Cited by 30 · see citing works
Generative AI in Language Education: Bridging Divide and Fostering Inclusivity
Weng, Z., & Fu, Y. (2025) · International Journal of Technology in Education · doi:10.46328/ijte.1056 · Cited by 31 · see citing works
“Up to this date, diverse review studies (e.g., Crompton, 2024; Jeon et al., 2023; Koc & Savas, 2024; Law, 2024; Liang et al., 2024; Ma et al., 2024; Wang et al., 2024; Yang & Li, 2024; Zhai & Wilbowo, 2022) on GenAI have been conducted, each with a different focus.”
The impact of generative artificial intelligence on students’ higher order thinking: evidence from a three-level meta-analysis
Nie, X., Tian, Y., Liu, M., Wu, D., & Guo, Y. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13735-x · Cited by 20 · see citing works
Emotional attachment as the key mediator: Expanding UTAUT2 to examine how perceived anthropomorphism in intelligent agents influences the sustained use of DouBao (Cici) among EFL learners
Zhang, Y., & Yu, Z. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13721-3 · Cited by 19 · see citing works
Investigating the capabilities of large language model-based task-oriented dialogue chatbots from a learner’s perspective
Lee, J. H., Shin, D., & Hwang, Y. (2024) · System · doi:10.1016/j.system.2024.103538 · Cited by 21 · see citing works
Applying Generative Artificial Intelligence to Task-based Language Teaching and Learning: A Systematic Review and Meta-analysis
Li, Y., Shadiev, R., & Chiu, T. K. F. (2025) · TechTrends · doi:10.1007/s11528-025-01140-7 · Cited by 37 · see citing works
A Systematic Review of the Role of Artificial Intelligence in Second Language Writing Education
Luo, Y., Xia, Y., & Lu, X. (2025) · Digital studies in language and literature · doi:10.1515/dsll-2025-0001 · Cited by 21 · see citing works
AI chatbot-assisted vocabulary learning: Relationships with self-regulation, motivation, and performance among Chinese private college students
Hao, C., Xu, W., Halim, H. B. A., & Hao, M. (2025) · Language Teaching Research · doi:10.1177/13621688251352595 · Cited by 20 · see citing works
Reconceptualizing the Role of the University Language Teacher in Light of Generative AI
Tutton, M., & Cohen, D. (2025) · Education Sciences · doi:10.3390/educsci15010056 · Cited by 12 · see citing works
Effects of self-regulated vocabulary learning with chatbots on incidental and collocational vocabulary learning and foreign language learning boredom
Jalambo, M. O., Çakmak, F., & Akhter, S. (2025) · Discover Education · doi:10.1007/s44217-025-00977-7 · Cited by 17 · see citing works
Can digital assistive technology interventions foster well-being among students with disabilities? A meta-analytic review
Pang, F., & Datu, J. A. D. (2025) · Educational Research Review · doi:10.1016/j.edurev.2025.100729 · Cited by 10 · see citing works
A Systematic Review on Generative AI in L2 Speaking Development From a Self‐Determination Theory Perspective
Lin, Y. C., Li, Y. K., Chen, X., & Chiu, T. K. F. (2025) · International Journal of Applied Linguistics · doi:10.1111/ijal.70091 · Cited by 9 · see citing works
Simulating the “cultural other”: the impact of GenAI interlocutors on EFL learners’ intercultural communicative competence, speaking anxiety, and learning engagement
Qu, Z., & Chen, L. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1799695 · Cited by 9 · see citing works
AI Gem: Context-Aware Transformer Agents as Digital Twin Tutors for Adaptive Learning
Kővári, A. (2025) · Computers · doi:10.3390/computers14090367 · Cited by 5 · see citing works
Perceived AI interactivity and Chinese EFL learners’ motivated learning behaviors: the moderating role of daily AI usage duration
Min, G., & Jiazhu, L. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1689099 · Cited by 6 · see citing works
Using artificial intelligence to support self-regulated learning: A systematic review of empirical research
Zhu, M., & Sari, A. R. (2026) · Learning and Individual Differences · doi:10.1016/j.lindif.2026.102917 · Cited by 6 · see citing works
Can ChatGPT as a generative AI tool enhance university student academic achievement?
Yu, Q., Yu, K., Yang, R., & Li, B. (2025) · Innovations in Education and Teaching International · doi:10.1080/14703297.2025.2605651 · Cited by 4 · see citing works
Student-AI collaboration patterns in project-based learning: Temporal and structural characteristics
Fei, X. (2026) · Journal of Research on Technology in Education · doi:10.1080/15391523.2026.2661697 · Cited by 3 · see citing works
Designing a Large Language Model Based Conversational Agent for Language Acquisition
Neis, N., Spleth, P., Kudlek, C., Zarnekow, R., & Winkelmann, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93976-1_15 · Cited by 2 · see citing works
AI-driven pathways to English proficiency: a short-term longitudinal study
Kim, E. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2570489 · Cited by 2 · see citing works
Effects of using data-driven and generative AI-assisted instructions on learning multiword constructions in written academic English
Rong, J., Deng, Y., & Liu, D. (2026) · Language Teaching Research · doi:10.1177/13621688251412525 · Cited by 2 · see citing works
Multiple sources and student engagement in the smart classroom: the mediating role of foreign language enjoyment under control-value theory
Wu, J. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1803799 · Cited by 2 · see citing works
Multimodal vocabulary instruction for English language learners: From interactive whiteboards to artificial intelligence
Martin, C. (2026) · European Journal of Education & Language Review · doi:10.20897/ejelr/18945 · Cited by 2 · see citing works
Corpus-driven and AI-assisted learning in teaching business academic writing: examining their efficacies and the predicting/moderating effects of learner variables
Wang, Q., Deng, Y., & Liu, D. (2025) · Computer Assisted Language Learning · doi:10.1080/09588221.2025.2589739 · Cited by 1 · see citing works
Effectiveness and implications of chatbot-supported language learning: a meta-analysis of global studies
Yang, Y. P., Song, Y., Hwang, G., & Jin, C. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2598061 · Cited by 1 · see citing works
Enhancing Verb‐Noun Collocation Learning among Chinese EFL Learners: A Comparison of Cognitive Linguistics‐Inspired, GenAI‐Assisted, and Traditional Instruction
Qin, J., Deng, Y., Li, D., & Liu, D. (2025) · International Journal of Applied Linguistics · doi:10.1111/ijal.70088 · Cited by 1 · see citing works
Explore the Gen-AI empowerment black box: a meta-analysis of the impact of Gen-AI on college students’ critical thinking
Jiang, X. (2026) · Interactive Technology and Smart Education · doi:10.1108/itse-01-2026-0004 · Cited by 1 · see citing works
Designing and Evaluating a Voice-Based Chatbot System for Second Language Speaking Fluency and Learner Engagement
Oh, E. Y., Choi, J., & Song, D. (2026) · Journal of Educational Technology Systems · doi:10.1177/00472395261435622 · Cited by 1 · see citing works
A decade of reflection and thematic review on artificial intelligence's impact on educational measurement
Xiong, J., & Li, F. (2026) · Educational Research Review · doi:10.1016/j.edurev.2026.100789 · Cited by 3 · see citing works
AI Chatbots and Dyslexia: How AI Chatbots Empower Students With Dyslexia to Master Arithmetic Word Problems
Wang, L., & Kuo, H. (2026) · Journal of Learning Disabilities · doi:10.1177/00222194261450136 · Cited by 1 · see citing works
Psychological and pedagogical effects of the implementation of artificial intelligence technologies in educational practice
Tokarchuk, Y. A., Rubtsova, O. V., & Tokarchuk, A. M. (2026) · Journal of Modern Foreign Psychology · doi:10.17759/jmfp.2026150201 · Cited by 1 · see citing works
Дидактичні особливості застосування чат-ботів на основі штучного інтелекту у формуванні іншомовної комунікативної компетентності майбутніх ерготерапевтів
Abakumov, A. (2026) · Grani · doi:10.15421/1726144 · Cited by 1 · see citing works
The role of neural networks in the development of communicative competence of technical science students
Troitskaya, Y. V. (2025) · Vestnik of Samara State Technical University Psychological and Pedagogical Sciences · doi:10.17673/vsgtu-pps.2025.1.3
CHATBOTS IN FOREIGN LANGUAGE SPEAKING INSTRUCTION: SCENARIOS, CHALLENGES, AND INTEGRATION
Kornytska, Y. (2025) · Věda a perspektivy · doi:10.52058/2695-1592-2025-6(49)-45-51
Voice-Based Chatbots for English Speaking Practice in Multilingual Low-Resource Schools: A Multi-Stakeholder Mixed-Methods StudyPreprint
Shashidhara, S., Ungar, L., & Guntuku, S. C. (2025) · doi:10.12688/verixiv.1351.1
Developing an AI-Driven Contextualized Short Video Learning System for EFL Speaking and Writing
Chang, H.-W., Liu, Y.-F., Hwang, W.-Y., & Chen, W.-C. (2025) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt64023.2025.00048
Dil Öğreniminde Yapay Zekâ Sohbet Robotları: ChatGPT Örneği
DEMİREL, A. (2025) · International Journal of Languages Education · doi:10.71084/ijlet.1734989
Chatbot Personalities in Digital Learning: Integrating Characteristics that Support Learning to Enhance Learner Engagement
Rahim, E. E. A., & Ismail, A. (2025) · International Conference on Information and Communication Technology for the Muslim World (ICT4M) · doi:10.1109/ict4m68001.2025.11363520
AI-Driven Feedback Through SpeakTalk: Supporting EFL Speaking Skill Development in Higher Education
Purba, S. W. D., Yang, J. J., Luthfi, M. I., Cendana, W., & Recard, M. (2025) · International Workshop on Artificial Intelligence and Education (WAIE) · doi:10.1109/waie67422.2025.11381221
AI Chatbots for Personalized Digital Marketing in Language Learning using the UTAUT2 model on Engagement and Continuous Usage IntentionPreprint
Toukabri, M. T., & Maherzi, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6883038
The impact of RAG chatbots on writing proficiency: A study of Vietnamese Grade 6 EFL students
Phuc, T. Q. B., Thangovich, N. Y., & Parveen, K. (2026) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2026.103135
Chatbot Versus Human Peer: Effects on College English as a Foreign Language Learners’ Interactional Competence Development
Meng, Y., Shen, C., Qian, X., & Wang, C. (2026) · Language Learning · doi:10.1111/lang.70044
SOHBET BOTLARINA YÖNELİK TUTUMLARIN VE KULLANIM ALIŞKANLIKLARININ İNCELENMESİ: PSİKOLOJİ LİSANS ÖĞRENCİLERİ ÜZERİNDE BİR ANKET ÇALIŞMASI
İyilikci, E. A., Yurt, H. M., Keysan, E. E., Ağaoğulları, N., Dallar, H., Güven, L. S., Özkan, M. S., Şahin, M., Günhan, Z., Baykuşak, E., et al. (2026) · Çukurova Üniversitesi Sosyal Bilimler Enstitüsü Dergisi · doi:10.35379/cusosbil.1647141
Effects of AI Chatbot-Led Versus Preservice Teacher-Led Interactive Reading on Engagement Trajectories and Training Outcomes
Zhou, W., Xiong, Y., Wang, R., Zhang, X., Liu, X., Lin, C., Zheng, H., Zheng, D., Xie, B., & Shu, Q. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2706899
Student–AI interaction: unveiling the impact of AI agents on vocabulary learning – A meta-analysis
Gao, K.-W., Shi, X., & Zhang, J. (2026) · IRAL - International Review of Applied Linguistics in Language Teaching · doi:10.1515/iral-2025-0382 · Cited by 2 · see citing works
Synthetic Dialogue Partners in Swedish Religious Education: Mediated Contact Through the Chatbot IRIS
Visuri, I., & Svensson, J. (2026) · Religion & Education · doi:10.1080/15507394.2026.2716456
Wie belastbar ist die KI-Bildungsforschung?
Friederichs, H. (2026) · doi:10.59350/twbre-sbr89
Wie belastbar ist die KI-Bildungsforschung?
Friederichs, H. (2026) · doi:10.59350/p0n7d-1qe84
Inteligencia artificial y aprendizaje en estudiantes universitarios
Frutos, F. M. O. (2026) · Scripta Scientia · doi:10.66201/ss.v1.50
2CARE: LLM-Based Classifier of Imitative Versus Creative Reasoning Achieving Near-Perfect Accuracy in Think-Aloud Transcripts
Vanhée, L., Boström, E., Lithner, J., Andersson, C., Palmberg, B., Winberg, M., Svahn, O., & Palm, T. (2026) · IEEE Access · doi:10.1109/access.2026.3733324
Multimodal AI avatars in L2 speaking: A time-lagged mediation analysis
Luan, J., Halim, H. B. A., & Saad, M. R. M. (2026) · System · doi:10.1016/j.system.2026.104163
Acoustic Fluency Patterns in AI-Mediated Task-Based French Instruction
Dinçer, S., Mavaşoğlu, M., & Kurt, Ç. (2026) · SAGE Open · doi:10.1177/21582440261492392
Situated Grammar Learning via the FOCUS Framework: Integrating AR, AI, and Gamification
Rahman, A. A. A. (2026) · International Conference on Software and Computer Applications (ICSCA) · doi:10.1145/3801619.3801626
Fragmented Yet Concentrated: Primary-Study Overlap in Meta-Analyses on Generative AI in Education
Otto, D. (2026) · Open Praxis · doi:10.55982/openpraxis.18.4.1297
Too Much Mind Extension? Towards a Virtue Ethics of the Extended Mind
Chlebowski, S., Ratajczyk, D., & Wyrwa, M. (2026) · Roczniki Filozoficzne · doi:10.18290/rf26743.6
Effects of speech-enabled LLM-based chatbots on L2 learning outcomes and affective domains: a multilevel meta-analysis
Lee, E., Lee, H., & Lee, J. H. (2026) · IRAL - International Review of Applied Linguistics in Language Teaching · doi:10.1515/iral-2026-0172
Design Framework for Older Adult Health and Fitness Technologies: A Proof of Concept Case Study (Preprint)
Tacca, C., Gomer, R., Thompson, I. M., Galvez, A. V., Freeman, C. T., Warner, M. B., Bincalar, A. D., Ng, A., & schraefel, M. C. (2026) · JMIR Formative Research · doi:10.2196/97010
Exploring language learning motivation through chatbot use: An action research study grounded in self‐determination theory
Villalobos-Buehner, M. (2026) · Foreign Language Annals · doi:10.1111/flan.70081

Data associated with the publication: Does chatting with chatbots improve language learning performance? A meta-analysis of chatbot-assisted language learning
Wang, F., Cheung, A. C. K., Neitzel, A. J., & Chai, C. S. (2024) · Research Data Repository, Duke University · doi:10.7281/t1/xol4br
Self-citation
Effectiveness and implications of chatbot-supported language learning: a meta-analysis of global studies
Yang, Y. P., Song, Y., Hwang, G., & Jin, C. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2598061 · Cited by 1 · see citing works
Effect of chatbot-assisted language learning: A meta-analysis
Zhang, S., Shan, C., Lee, J. S. Y., Che, S., & Kim, J. H. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11805-6 · Cited by 103 · see citing works
Effectiveness of Chatbots in Improving Language Learning: A Meta‐Analysis of Comparative Studies
Lyu, B., Lai, C., & Guo, J. (2024) · International Journal of Applied Linguistics · doi:10.1111/ijal.12668 · Cited by 67 · see citing works
The effect of chatbots on learning: a meta-analysis of empirical research
Alemdağ, E. (2023) · Journal of Research on Technology in Education · doi:10.1080/15391523.2023.2255698 · Cited by 78 · see citing works

Authors: Wang, F., Cheung, A. C. K., Neitzel, A. J., & Chai, C. S.
Affiliations: Chinese University of Hong Kong (HK)
Johns Hopkins University (US)
Corresponding author: Wang, Feifei
ORCID: Wang, Feifei
Cheung, Alan C. K.
Neitzel, Amanda J.
Chai, Ching Sing
Venue: Review of Educational Research
Publisher: American Educational Research Association (AERA)
Published: 2024-06-14
Volume: 95
Issue: 4
Pages: 623-660
ISSN: 0034-6543 (ISSN-L)
1935-1046
DOI: 10.3102/00346543241255621
OpenAlex: W4399681132
Semantic Scholar: febd1f1af796db3ea56a518b9c3ce6697598ca04
Cited by: 64
References: 93
Citations by year: 2026: 31 · 2025: 30 · 2024: 3
FWCI: 14.84
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Review of Educational Research (publishedVersion)
The HKU Scholars Hub (University of Hong Kong) (submittedVersion)
Topics: Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
Physical Sciences › Computer Science › Information Systems › English Language Learning and Teaching
Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords: chatbots, chatbot-assisted language learning, ChatGPT, language learning performance, conversation practice, interface design, learner characteristics, language proficiency levels, educational attainment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Does Chatting with Chatbots Improve Language Learning Performance? A Meta-Analysis of Chatbot-Assisted Language Learning

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks