CoChat

Personalized language learning with an LLM chatbot: effects of immediate vs. delayed corrective feedback

Open access
M. Kamelabad, Alireza, Turano, Beatrice, Lundin, Mattias, Skantze, Gabriel · 2026 · Frontiers in Education

The DOI matches its Crossref record; 115 of the 116 references it cites were found in a registry or an index; none of the 97 sources its claims cite is checked for support yet.

Abstract

The emergence of Large Language Models (LLMs) has opened new possibilities for language learning through conversational interaction with chatbots. Yet, little empirical evidence exists on how students experience such interactions and how corrective feedback should be provided. Research suggests that immediate corrective feedback is generally more effective than delayed feedback. Nevertheless, learners' perception of this effectiveness and their preferences for feedback timing, particularly in the domain of Computer-Assisted Language Learning (CALL), remain underexplored. This study investigates the feasibility of providing immediate feedback and examines the impact of feedback timing on user experience and grammar learning gains in English. An in-the-wild experiment was conducted with 66 L2 English learners, who integrated chatbot sessions into their English course as an extracurricular activity over one semester. Participants were randomly assigned to two groups receiving feedback either during or after the conversation. Findings reveal no significant difference in learning gains, but immediate feedback enhanced user experience, leading to overall positive perceptions of the chatbot. Additionally, we explore users' perceptions of the chatbot's social role and personality, offering a roadmap for future enhancements. These results provide valuable insights into the potential of LLMs and chatbots for language learning.

Overview

CoChat Summary

TL;DR

In an in-the-wild experiment with 66 L2 English learners (54 analyzed), an LLM-powered chatbot provided either immediate or delayed corrective feedback over 12 sessions. Immediate feedback improved perceived effectiveness and user experience, but the study found no significant difference in grammar learning gains between feedback timing conditions.

Key findings

  • No significant difference in grammar learning outcomes between immediate corrective feedback (ICF) and delayed corrective feedback (DCF): language assessment scores were ICF x = 3.14 vs. DCF x = 2.18, t(47.99) = 1.35, p = 0.092, Cohen’s d = 0.37; overall participants corrected x = 2.62 of 15 mistakes. A GLMM found no significant relationship between error counts and condition, session number, or error type.
  • Immediate feedback improved perceived effectiveness: post-questionnaire ICF x = 3.65 vs. DCF x = 3.08, t(47.96) = 2.01, p = 0.0495, Cohen’s d = 0.56; ICF showed a significant paired increase (x = 1.48, t(22) = 2.63, p = 0.015, d = 0.68), while DCF did not (x = 0.43, t(29) = 1.15, p = 0.259, d = 0.23). Pre-test effectiveness did not differ between groups (t(44.28) = 0.25, p = 0.8, d = 0.07).
  • Overall pre-post effectiveness ratings improved: sign test found 22 of 31 non-zero changes were positive, p = 0.0294; estimated probability of positive change was 70.97%, 95% CI [51.96%, 85.78%]. 72.22% (39/54) reported learning something in English, χ²(1) = 9.79, p = 0.0017, 95% CI [58.14%, 83.14%].
  • The chatbot was perceived anthropomorphically: 56% considered it male, 31% assigned no gender; mean assigned age was 30.94 (s = 12.59); social role choices were 40% “a random person,” 27.3% “friend,” 10.9% “teacher,” and 21.8% “a mobile application.” Experience was moderately positive (x = 3.69, s = 0.72) and politeness high (x = 4.52, s = 0.75); determiners were the most common error type.

Methods

  • In-the-wild between-subject experiment: 66 bachelor students enrolled, 54 completed and were analyzed (24 ICF, 30 DCF); ages 18–23 (x = 20.59, s = 0.98), 43 males and 11 females, mostly native Czech speakers, with self-assessed English levels from A2 to C2. Participants completed 12 chatbot sessions over 4 weeks (3 per week), each until 1,000 characters, outside school.
  • The chatbot was a web application using OpenAI GPT-3 text-davinci-003 API, Python/Flask/JavaScript, and Gramformer for grammatical error detection/correction. Feedback types were explicit recast, negative evidence, and combined feedback; ICF corrected the first mistake immediately during conversation, while DCF accumulated errors and showed a summary report after the conversation.
  • Measures included pre/post questionnaires on effectiveness, ranking, chatbot personality/social role, experience, and politeness; a language assessment of 25 sentences (about 15 containing the participant’s own errors) scored 1 point per corrected mistake; and logged error counts analyzed with a GLMM. Analyses used R 4.3.2 with sign test, independent/paired t-tests, one-sample proportions test, and p < 0.05.

Limitations

  • Paper-stated: no comprehensive pre/post language test; learning was measured via participants’ own mistakes, self-reported proficiency may bias results; varying session topics and external factors such as tiredness or mood may affect error counts or attention; no control group; large dropout (12 of 66); small sample limits generalization; one-month duration may be insufficient; asking participants to correct their own errors may have minimal priming; time was not controlled, though the paper argues this was an alternative design choice rather than necessarily a limitation.
  • Paper-stated future needs: more controlled designs, larger samples, a control group, within-subject designs, focus on specific structures such as determiners, diverse learner backgrounds, long-term engagement, and more advanced LLMs.
  • Evident from the text: attrition produced imbalanced groups (24 ICF vs. 30 DCF); the paper notes N = 54 may have been underpowered to detect subtle grammar acquisition differences; open-domain conversation lacked structured output practice, which the paper suggests may explain comparable gains despite differing perceived salience.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/feduc.2026.1703664

Cites

115 of 116 references verified0 citation errors · 1 not checkable

Statistics

5 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Figures

Image 1
CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Image 5
CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 7 Not valid · 73 Not checked · 110 Own results and interpretations

50 more claims

Create a free account to see all 190 claims and to verify the 82 references still open.

See all claims — free account
  • Probably not a substantial claim. 111
  • It may state two separate claims as one. 62

Create a free account to see these claims.

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(47.99) = 1.35reported p = .092 · recomputed p = .183

    “t(47.99)= 1.35, p = .092”

  • Consistentt(44.28) = 0.25reported p = .8 · recomputed p = .804

    “t(44.28)= 0.25, p = .8”

  • Consistentt(47.96) = 2.01reported p = .0495 · recomputed p = .050

    “t(47.96)= 2.01, p = .0495”

  • Consistentt(22) = 2.63reported p = .015 · recomputed p = .015

    “t(22) = 2.63, p = .015”

  • Consistentt(29) = 1.15reported p = .259 · recomputed p = .260

    “t(29) = 1.15, p = .259”

5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 116 references

Research works printed but not linked: 5

  1. Haines, 1995, For and against: pairwork, Modern Engl. Teach, 4, 55
  2. AI, expert or peer? – examining the impact of perceived feedback source on pre-service teachers feedback Perception and uptake Jacobsen L. J. Mertens U. Jansen T. Weber K. E. arXiv 2025
  3. Kim, 2019, Future English learning: chatbots and artificial intelligence, Multimedia-Assis. Lang. Learn, 22, 32, 10.15702/mall.2019.22.3.32 · doi:10.15702/mall.2019.22.3.32
  4. 2023, R: A Language and Environment for Statistical Computing
  5. Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9

Cited by

See all 13

Metadata

Authors
M. Kamelabad, A., Turano, B., Lundin, M., & Skantze, G.
Affiliations
  • Division of Speech, Music, and Hearing, KTH Royal Institute of Technology
  • University of Trento (IT)
  • KTH Royal Institute of Technology (SE)
Venue
Frontiers in Education
Published
2026-02-24
Volume
11
ISSN
2504-284X (ISSN-L)
OpenAlex
W7131254148
Cited by
13
References
113
Citations by year
2026: 12
FWCI
82.84
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Education (publishedVersion, cc-by)
  • DiVA (Digitala Vetenskapliga Arkivet) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • Open Science Framework
Funders
  • H2020 Marie Skłodowska-Curie Actions (857897)
  • HORIZON EUROPE Marie Sklodowska-Curie Actions (857897)
Topics
  • Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Social Sciences › Education › English Language Learning and Teaching
Keywords
large language models, chatbots, corrective feedback, feedback timing, grammar learning, English as a second language, computer-assisted language learning, user experience, learner perceptions, chatbot personality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Personalized language learning with an LLM chatbot: effects of immediate vs. delayed corrective feedback. https://app.cochat.ai/papers/10.3389/feduc.2026.1703664
The paper
M. Kamelabad, A., Turano, B., Lundin, M., & Skantze, G. (2026). Personalized language learning with an LLM chatbot: effects of immediate vs. delayed corrective feedback. Frontiers in Education. https://doi.org/10.3389/feduc.2026.1703664

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.