The DOI matches its Crossref record, with a small difference in title or year; 32 of the 32 references it cites were found in a registry or an index.
The effect of chatbots on learning: a meta-analysis of empirical research
Abstract
This meta-analysis aimed to comprehensively review empirical studies on the effect of chatbots on learning and quantitatively synthesize their findings to produce an overall effect size. Searching several databases yielded 28 eligible reports with 31 individual effect sizes. The results revealed a significant and medium effect (g = .48) of chatbots on learning. Further analyses indicated four significant moderators: the type of instruction in the comparison group, experimental duration, chatbot type, and chatbot tasks. The highest effect sizes emerged when the comparison group had no specific support, the experiment lasted only one session, and chatbots were task-focused and took care of frequently asked questions. These results suggest that chatbots can be more effective in certain cases within their overall contribution area to learning.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Secondly, the impact of GenAI varies significantly across different disciplines, a finding that diverges from Alemdag (2023).”
From Effects of Generative Artificial Intelligence on K-12 and Higher Education Students’ Learning Outcomes: A Meta-Analysis (2025) · cites this paper as Alemdag (2023)
Citations
Cites
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262 · cited 2×
“A sensitivity analysis was also conducted to evaluate the robustness of the synthesized findings (Page et al., 2021).”
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4102_1 · Cited by 7,776 · cited 1×
“Learners with low self-regulation skills and prior knowledge can encounter problems in such environments and experience cognitive load while directing their learning processes with minimal guidance (Kirschner et al., 2006).”
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 4×
“Despite the latest advancements in chatbot technology that can perform various natural language tasks, Kasneci et al. (2023) underscore that “they can only serve as assistive tools to human learners and educators and cannot replace the teacher” (p. 5).”
InfluentialSystematic Reviews in the Social Sciences: A Practical Guide
Petticrew, M., & Roberts, H. (2006) · doi:10.1002/9780470754887 · Cited by 3,586
Outlier and influence diagnostics for meta-analysis
Viechtbauer, W., & Cheung, M. W.-L. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.11 · Cited by 2,949
Analysing data and undertaking meta‐analyses
Deeks, J. J., Higgins, J. P., Altman, D. G., & on behalf of the Cochrane Statistical Methods Group (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch10 · Cited by 2,672 · cited 1×
“They are especially helpful in resolving controversies concerning contradictory studies and quantitatively identifying the reasons for different results (Deeks et al., 2022).”
Instructional Design Theories and Models
(1983) · doi:10.4324/9780203824283 · Cited by 2,192
Estimating effect sizes from pretest-posttest-control group designs
Morris, S. B. (2008) · Organizational Research Methods · doi:10.1177/1094428106291059 · Cited by 2,133
What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00237-x · Cited by 2,003 · cited 1×
“Even after new chatbots were trained with a large amount of data, similar concerns were reported, and new issues (e.g. plagiarism, cheating, and different responses to the same questions) arose (Farrokhnia et al., 2023; Tlili et al., 2023).”
Research works printed but not linked: 56
- Bibauw, S., Van den Noortgate, W., François, T., & Desmet, P. (2022). Dialogue systems for language learning: A meta-analysis. Language Learning & Technology, 26(1), 1–25. Retrieved from https://hdl.handle.net/10125/73488
- 10.1002/9780470743386 · DOI · Paper page
- 10.1007/s10257-020-00471-7 · DOI
- *Chang, C. Y., Kuo, S. Y., & Hwang, G. H. (2022). Chatbot-facilitated nursing education. Educational Technology & Society, 25(1), 15–27.
- 10.1163/23641177-bja10062 · DOI
- 10.1080/10494820.2023.2172044 · DOI · Paper page
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
- 10.1109/EITT.2018.00054 · DOI
- 10.1080/14703297.2023.2195846 · DOI · Paper page
- 10.1177/07356331221077901 · DOI · Paper page
- 10.1016/j.chb.2017.05.045 · DOI · Paper page
- 10.7238/elc.chatbots.2018 · DOI
- *Graesser, A. C., Jackson, G. T., Mathews, E. C., Mitchell, H. H., Olney, A., & Ventura, M. (2003). Why/AutoTutor: A test of learning gains from a physics tutor with natural language dialog. In R. Alterman & D. Hirsh (Eds.), Proceedings of the 25th Annual Conference of the Cognitive Science Society (pp. 1–5) Cognitive Science Society.
- 10.1016/j.compedu.2021.104395 · DOI
- Hattie, J. A. C. (2009). Visible learning: A synthesis of 800 meta-analyses relating to achievement. Routledge.
- 10.1007/s11412-020-09333-3 · DOI
- 10.1016/B978-0-08-057065-5.50010-5 · DOI · Paper page
- 10.1136/bmj.327.7414.557 · DOI · Paper page
- 10.1145/2493432.2493502 · DOI
- 10.30191/ETS.202301_26(1).0016 · DOI
- 10.1111/jcal.12610 · DOI · Paper page
- 10.1007/s10639-023-11602-1 · DOI
- 10.1080/15391523.2022.2142873 · DOI · Paper page
- *Kerly, A., Ellis, R., & Bull, S. (2009). Conversational agents in e-learning. In T. Allen, R. Ellis, & M. Petridis (Eds.), Applications and innovations in intelligent systems XVI (pp. 169–182). Springer.
- *Kim, N. Y. (2016). Effects of voice chat on EFL learners’ speaking ability according to proficiency levels. Multimedia-Assisted Language Learning, 19(4), 63–88.
- *Kim, N. Y. (2018a). A study on chatbots for developing Korean college students’ English listening and reading skills. Journal of Digital Convergence, 16(8), 19–26.
- *Kim, N. Y. (2018b). Chatbots and Korean EFL students’ English vocabulary learning. Journal of Digital Convergence, 16(2), 1–7.
- 10.18823/asiatefl.2021.18.1.10.161 · DOI
- 10.1016/j.pec.2016.10.024 · DOI
- 10.2307/2529310 · DOI · Paper page
- 10.21559/aellk.2022.48.1.011 · DOI
- 10.19173/irrodl.v20i4.4224 · DOI
- *Lin, M. P.-C., & Chang, D. (2020). Enhancing post-secondary writers’ writing skills with a chatbot: A mixed-method classroom study. Educational Technology & Society, 23(1), 78–92.
- Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. SAGE.
- 10.3390/app12073239 · DOI
- 10.1109/ICTA.2017.8336020 · DOI
- Nghi, T. T., Phuc, T. H., & Thang, N. T. (2019). Applying Al chatbot for teaching a foreign language: An empirical research. International Journal of Scientific & Technology Research, 8(12), 897–902.
- 10.1016/j.caeai.2021.100033 · DOI · Paper page
- 10.1002/cae.22326 · DOI · Paper page
- 10.1037/0033-2909.86.3.638 · DOI · Paper page
- 10.1145/3397481.3450648 · DOI
- 10.4135/9781483398105 · DOI · Paper page
- 10.1002/jrsm.1260 · DOI · Paper page
- *Tegos, S., & Demetriadis, S. (2017). Conversational agents improve peer learning through building on prior knowledge. Educational Technology & Society, 20(1), 99–111.
- 10.1007/s11412-016-9246-2 · DOI
- 10.2304/elea.2013.10.2.161 · DOI
- 10.1186/s41239-021-00269-8 · DOI
- 10.1016/j.stueduc.2019.100828 · DOI
- 10.1016/j.tate.2019.06.024 · DOI
- 10.1145/3411764.3445781 · DOI · Paper page
- 10.5465/AMBPP.2018.15903abstract · DOI · Paper page
- 10.1016/j.compedu.2020.104059 · DOI
- 10.1111/bjet.12758 · DOI
- 10.1177/0735633120952067 · DOI · Paper page
- 10.1007/s10639-023-11805-6 · DOI · Paper page
- 10.1080/10494820.2023.2202704 · DOI
Cited by
Liu, X., Guo, B., He, W., & Hu, X. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251329185 · Cited by 89
Salinas-Navarro, D. E., Vilalta-Perdomo, E., Michel-Villarreal, R., & Montesinos, L. (2024) · Interactive Technology and Smart Education · doi:10.1108/itse-12-2023-0236 · Cited by 84
Xu, Y., Zhu, J., Wang, M., Qian, F., Yang, Y., & Zhang, J. (2024) · Applied Sciences · doi:10.3390/app14156418 · Cited by 60
The Impact of ChatGPT on Students' Academic Achievement: A Meta‐Analysis
Liu, Z., Zuo, H., & Lu, Y. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70096 · Cited by 44
Zhu, Y., Zhu, C., Wu, T., Wang, S., Zhou, Y., Chen, J., Wu, F., & Li, Y. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-12898-3 · Cited by 33
Metadata
- Authors
- Alemdag, E.
- Affiliations
- Technische Universität Dresden
- ORCID
- Alemdag, Ecenaz
- Venue
- Journal of Research on Technology in Education
- Publisher
- Informa UK Limited
- Published
- 2023-09-12
- Volume
- 57
- Issue
- 2
- Pages
- 459-481
- ISSN
- 1539-1523
- 1945-0818
- Cited by
- 67
- References
- 32
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The effect of chatbots on learning: a meta-analysis of empirical research. https://app.cochat.ai/papers/10.1080/15391523.2023.2255698
- The paper
- Alemdag, E. (2025). The effect of chatbots on learning: a meta-analysis of empirical research. Journal of Research on Technology in Education. https://doi.org/10.1080/15391523.2023.2255698
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
