Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index; 10 of the 56 sources its claims cite were checked for support.

Impact of ChatGPT on ESL students’ academic writing skills: a mixed methods intervention study

Open access
Mahapatra, Santosh · 2024 · Smart Learning Environments

Abstract

A mixed methods intervention study reporting significant positive impact of ChatGPT as a formative feedback tool on undergraduate ESL students' academic writing skills and positive perceptions.

Overview

CoChat Summary

TL;DR

This mixed methods intervention study found that using ChatGPT as a formative feedback tool significantly improved undergraduate ESL students’ academic writing skills, with the experimental group outperforming the comparison group on post-test and delayed post-test. Students’ focus group perceptions of ChatGPT’s impact were also overwhelmingly positive, though some concerns about dependence and reduced creativity were reported.

Key findings

  • The experimental group’s writing scores improved significantly across pre-test, post-test, and delayed post-test: one-way RM ANOVA, F(2, 72) = 330.704, p = 5.146e-37, η² = 0.902. Bonferroni comparisons showed pre-test to post-test p < 0.001, d = -3.300; pre-test to delayed post-test p < 0.001, d = -3.898; post-test to delayed post-test p < 0.01, d = -0.598.
  • The experimental group outperformed the comparison group on the post-test: t(70) = -5.643, p = 3.297e-7, with EG M = 19.216, SD = 2.485 vs. CG M = 16.371, SD = 1.695. The difference remained significant on the delayed post-test: t(70) = -9.371, p = 5.544e-14, with EG M = 20.419, SD = 2.575 vs. CG M = 15.400, SD = 1.897.
  • Focus group data were coded under content, organization, and grammar. Positive codes included generation of more ideas (22 references), focused information on the topic (21), promotion of learner autonomy (19), promotion of collaboration (15), faster writing speed (13), creation of strong topic sentences (13), easy to stay focused (22), strong connection between main idea and supporting details (17), improving accuracy in sentence structure (22), and reliable source of grammar (21). Negative codes included less motivation to think (4), more machine dependence (3), imposing a pattern on writing (2), hindering creative organization (2), reduction in attention to grammatical accuracy (5), and increase in machine-dependence (3).

Methods

  • Mixed methods intervention design with a quasi-experimental component. Participants were first-year science and engineering ESL students at an elite private university in India, aged 18–19. Initially 78 EG and 56 CG students consented; final reported data included 35 EG and 37 CG students who attended all six intervention hours and took the pre-, post-, and delayed post-tests. Five students participated in all three focus group discussions.
  • The experimental group received one-hour training on using ChatGPT for self- and peer assessment, then was taught process, comparison, and cause-effect writing for six hours with ChatGPT used for SA and PA. Data came from three writing tests (process, comparison, cause-effect) scored with rubrics covering content, organization, grammar, and vocabulary; inter-rater Cohen’s Kappa was >0.8. A delayed post-test was conducted almost two months after the post-test.
  • Quantitative analysis used one-way RM ANOVA, Bonferroni post-hoc tests, independent samples t-tests, Levene’s test, Shapiro-Wilk test, and Grubb’s test. Qualitative FGD transcripts were coded using a phronetic iterative approach, and quantitative and qualitative findings were triangulated.

Limitations

  • The paper states that the study focused on only three writing genres, the intervention lasted only six hours, and student artefacts were not included. It notes that the intact classroom prevented going beyond the prescribed syllabus, focusing on more components, or extending the intervention, and that privacy and copyright issues made artefact collection difficult. It also states that argumentative writing was ignored, that generalization should be carefully worded, and that the absence of artefacts hurts the validity of claims.
  • Evident from the text: the final analytic sample was relatively small (35 EG and 37 CG), and the qualitative perception data came from only five students who participated in all three FGDs, limiting breadth.
  • Evident from the text: the study was conducted in a single course and institution with first-year science and engineering students, so findings may not transfer to other ESL/EFL populations, genres, or instructional contexts.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s40561-024-00295-9

Cites

56 of 57 references verified1 citation errors · 0 not checkable

Statistics

Figures

Fig. 2 Intervention plan
Fig. 2 Intervention planCC-BY · From the PDF

Verification

Claims the paper makes

9 Valid · 3 Not valid · 31 Not checked · 63 Own results and interpretations

27 more claims

Create a free account to see all 106 claims and to verify the 5 references still open.

See all claims — free account
  • Probably not a substantial claim. 62
  • It may state two separate claims as one. 37

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentt(70) = -5.643reported p = 3.297e-7 · recomputed p < .001

    “t(70) = -5.643,p = 3.297e-7”

  • Consistentt(70) = -9.371reported p = 5.544e-14 · recomputed p < .001

    “t(70) = -9.371,p = 5.544e-14”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 51 references

Research works printed but not linked: 2

  1. Mathur, M., & Mahapatra, S. (2022). Impact of ePortfolio assessment as an instructional strategy on students’ academic speaking skills: An experimental study. CALL-EJ, 23(3), 1–23.
  2. Tai, A. M. Y., Meyer, M., Varidel, M., Prodan, A., Vogel, M., Iorfino, F., & Krausz, R. M. (2023). Exploring the potential and limitations of ChatGPT for academic peer-reviewed writing: Addressing linguistic injustice and ethical concerns. Journal of Academic Language and Learning, 17(1), T16–T30.

Cited by

See all 398

Metadata

Authors
Mahapatra, S.
Affiliations
Birla Institute of Technology and Science - Hyderabad Campus (IN)
Corresponding author
Mahapatra, Santosh
Venue
Smart Learning Environments
Published
2024-02-13
Volume
11
Issue
1
ISSN
2196-7091 (ISSN-L)
OpenAlex
W4391780419
Cited by
398
References
51
Citations by year
2026: 192 · 2025: 154 · 2024: 52
FWCI
623.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Smart Learning Environments (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
  • Social Sciences › Education › Artificial Intelligence in Education
  • Social Sciences › Education › English Language Learning and Teaching
Keywords
ChatGPT, academic writing skills, ESL learners, formative feedback, generative AI, automated writing evaluation, dialogic feedback, student perceptions, writing instruction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Impact of ChatGPT on ESL students’ academic writing skills: a mixed methods intervention study. https://app.cochat.ai/papers/10.1186/s40561-024-00295-9
The paper
Mahapatra, S. (2024). Impact of ChatGPT on ESL students’ academic writing skills: a mixed methods intervention study. Smart Learning Environments. https://doi.org/10.1186/s40561-024-00295-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.