The DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index; 10 of the 56 sources its claims cite were checked for support.
Impact of ChatGPT on ESL students’ academic writing skills: a mixed methods intervention study
Open accessAbstract
A mixed methods intervention study reporting significant positive impact of ChatGPT as a formative feedback tool on undergraduate ESL students' academic writing skills and positive perceptions.
Overview
CoChat Summary
TL;DR
This mixed methods intervention study found that using ChatGPT as a formative feedback tool significantly improved undergraduate ESL students’ academic writing skills, with the experimental group outperforming the comparison group on post-test and delayed post-test. Students’ focus group perceptions of ChatGPT’s impact were also overwhelmingly positive, though some concerns about dependence and reduced creativity were reported.
Key findings
- The experimental group’s writing scores improved significantly across pre-test, post-test, and delayed post-test: one-way RM ANOVA, F(2, 72) = 330.704, p = 5.146e-37, η² = 0.902. Bonferroni comparisons showed pre-test to post-test p < 0.001, d = -3.300; pre-test to delayed post-test p < 0.001, d = -3.898; post-test to delayed post-test p < 0.01, d = -0.598.
- The experimental group outperformed the comparison group on the post-test: t(70) = -5.643, p = 3.297e-7, with EG M = 19.216, SD = 2.485 vs. CG M = 16.371, SD = 1.695. The difference remained significant on the delayed post-test: t(70) = -9.371, p = 5.544e-14, with EG M = 20.419, SD = 2.575 vs. CG M = 15.400, SD = 1.897.
- Focus group data were coded under content, organization, and grammar. Positive codes included generation of more ideas (22 references), focused information on the topic (21), promotion of learner autonomy (19), promotion of collaboration (15), faster writing speed (13), creation of strong topic sentences (13), easy to stay focused (22), strong connection between main idea and supporting details (17), improving accuracy in sentence structure (22), and reliable source of grammar (21). Negative codes included less motivation to think (4), more machine dependence (3), imposing a pattern on writing (2), hindering creative organization (2), reduction in attention to grammatical accuracy (5), and increase in machine-dependence (3).
Methods
- Mixed methods intervention design with a quasi-experimental component. Participants were first-year science and engineering ESL students at an elite private university in India, aged 18–19. Initially 78 EG and 56 CG students consented; final reported data included 35 EG and 37 CG students who attended all six intervention hours and took the pre-, post-, and delayed post-tests. Five students participated in all three focus group discussions.
- The experimental group received one-hour training on using ChatGPT for self- and peer assessment, then was taught process, comparison, and cause-effect writing for six hours with ChatGPT used for SA and PA. Data came from three writing tests (process, comparison, cause-effect) scored with rubrics covering content, organization, grammar, and vocabulary; inter-rater Cohen’s Kappa was >0.8. A delayed post-test was conducted almost two months after the post-test.
- Quantitative analysis used one-way RM ANOVA, Bonferroni post-hoc tests, independent samples t-tests, Levene’s test, Shapiro-Wilk test, and Grubb’s test. Qualitative FGD transcripts were coded using a phronetic iterative approach, and quantitative and qualitative findings were triangulated.
Limitations
- The paper states that the study focused on only three writing genres, the intervention lasted only six hours, and student artefacts were not included. It notes that the intact classroom prevented going beyond the prescribed syllabus, focusing on more components, or extending the intervention, and that privacy and copyright issues made artefact collection difficult. It also states that argumentative writing was ignored, that generalization should be carefully worded, and that the absence of artefacts hurts the validity of claims.
- Evident from the text: the final analytic sample was relatively small (35 EG and 37 CG), and the qualitative perception data came from only five students who participated in all three FGDs, limiting breadth.
- Evident from the text: the study was conducted in a single course and institution with first-year science and engineering students, so findings may not transfer to other ESL/EFL populations, genres, or instructional contexts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 3 Not valid · 31 Not checked · 63 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Since it can automatically train itself and learn from previous conversations (Chan & Hu, 2023)”
§ Impact of ChatGPT on ESL students' academic writing skills: a mixed methods intervention study > Literature review
What the sources say
- Could not verifycites Chan & Hu, 2023 sourceExperimental — accuracy not yet measured
The “Generative Artificial Intelligence” section (p. 2) says ChatGPT was pre-trained on text and can maintain coherent conversations. Searches for “learn from previous conversations” and “automatically train itself” found no passage saying it trains itself or learns from conversations; pre-training alone does not establish either claim. (LLM, conf=0.77)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“In terms of organization, the perceived impact confirms the findings of Allagui (2023) and Marzuki et al. (2023), who highlight the help with the organization of content in their study.”
§ Impact of ChatGPT on ESL students' academic writing skills: a mixed methods intervention study > Students' positive outlook
What the sources say
- Not checked — no full text of the sourcecites Allagui (2023) source
- Supportedcites Marzuki et al. (2023) source
In section 5 (Discussion), Marzuki et al. report that all four participating teachers recognized AI writing tools as helping the clarity and logical progression of students’ writing. This supports the citing sentence’s claim about a perceived benefit for organizing content; the supplied passage does not give a page number. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A significant difference was observed between the post-test and delayed post-test scores (p < 0.01, d = -0.598)”
§ Impact of ChatGPT on ESL students' academic writing skills: a mixed methods intervention study > Positive impact of ChatGPT as a feedback tool
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In their comments on the impact of ChatGPT as a feedback tool on organization in writing, it was an overwhelming claim that ChatGPT made it easier to stay focused on the topic when writing.”
§ Impact of ChatGPT on ESL students' academic writing skills: a mixed methods intervention study > It might be a concern that my dependence might discourage me to do things on my own when writing. What if I won't want to write on my own? It can be scary, but I don't know. (S5, FGD 3)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Though the CG and the EG students demonstrated improved writing skills during the period under consideration, the EG's performance was significantly better than their CG peers across two post-intervention tests.”
§ Impact of ChatGPT on ESL students' academic writing skills: a mixed methods intervention study > Positive impact on writing skills
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
27 more claims
Create a free account to see all 106 claims and to verify the 5 references still open.
- Probably not a substantial claim. 62
- It may state two separate claims as one. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Intervention studies have shown that using ChatGPT as a formative feedback tool can enhance writing accuracy, organization, and learner engagement while reducing anxiety (Khampusaen, 2025; Mahapatra, 2024).”
From Perceptions vs. practices: academic integrity and actual ChatGPT use among EFL students (2026) · cites this paper as Mahapatra, 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“align with the intervention results of (Khampusaen 2025) and (Mahapatra 2024), who reported that ethically integrated AI feedback enhanced writing fluency and confidence.”
From Perceptions vs. practices: academic integrity and actual ChatGPT use among EFL students (2026) · cites this paper as Mahapatra 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the realm of early intervention, positive outcomes on writing performance [5], [6], self-regulated learning and motivation [6] have been reported when LLMs are systematically integrated into instruction”
From Generative AI as an Automated Written Corrective Feedback Provider in EFL Academic Writing: A Mixed-Methods Investigation of Accuracy Gains, Feedback Quality, and Learner Engagement (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The generally positive perception of ChatGPT's support in writing, particularly in grammar correction, idea generation, and coherence.Aligned with the findings of Song and Song (2023), Mahapatra (2024), and Barrot (2023), who observed that ChatGPT enhanced writing fluency and lexical richness.”
From An Investigation into the Usage of ChatGPT in Academic Writing by EFL University Students in North and East Syria: A Cross-Sectional Study (2025) · cites this paper as Mahapatra (2024)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Similarly, Mahapatra (2024) noted the significant positive effect of using ChatGPT as a feedback tool on students' writing skills.”
From An Investigation into the Usage of ChatGPT in Academic Writing by EFL University Students in North and East Syria: A Cross-Sectional Study (2025) · cites this paper as Mahapatra (2024)
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentt(70) = -5.643reported p = 3.297e-7 · recomputed p < .001
“t(70) = -5.643,p = 3.297e-7”
- Consistentt(70) = -9.371reported p = 5.544e-14 · recomputed p < .001
“t(70) = -9.371,p = 5.544e-14”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Impact of ChatGPT on ESL students' academic writing skills: a mixed methods intervention study > Participants
“Thus, 35 students from the EG and 37 students from the CG were included in the study.”
§ Impact of ChatGPT on ESL students' academic writing skills: a mixed methods intervention study > Conclusion
“| Score | CG | 35 | 15.400 | 1.897 | 0.321 | | | EG | 37 | 20.419 | 2.575 | 0.423 |”
Citations
Cites
Designing and Conducting Mixed Methods Research
Robinson, P. (2007) · Australian and New Zealand Journal of Public Health · doi:10.1111/j.1753-6405.2007.00096.x · Cited by 23,258
Designing and Conducting Mixed Methods Research, 3rd ed.
Creswell, J. W. (2018) · Cited by 12,959
Qualitative Research Methods: Collecting Evidence, Crafting Analysis, Communicating Impact
Tracy, S. J. (2012) · Cited by 4,112
Students' voices on generative ai: perceptions, benefits, and challenges in higher education
Chan, C. K. Y., & Hu, W. (2023) · arXiv · doi:10.48550/arxiv.2305.00290 · Cited by 2,504 · cited 1×
“Since it can automatically train itself and learn from previous conversations (Chan & Hu, 2023), students can receive tailored feedback suitable for individual needs.”
BackgroundStudents’ voices on generative AI: perceptions, benefits, and challenges in higher education
Chan, C. K. Y., & Hu, W. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00411-8 · Cited by 2,301 · cited 1×
“Since it can automatically train itself and learn from previous conversations (Chan & Hu, 2023), students can receive tailored feedback suitable for individual needs.”
BackgroundLanguage Learning with Technology: Perspectives from Asia
(2021) · Springer eBooks · doi:10.1007/978-981-16-2697-5 · Cited by 2,206
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806 · cited 4×
“Many existing studies utilized these tools for formative purposes (Rudolph et al., 2023) and found similar results.”
BackgroundMethodResultInfluentialChatGPT for Language Teaching and Learning
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023) · RELC Journal · doi:10.1177/00336882231162868 · Cited by 1,371
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099 · cited 1×
“It has been reported as capable of producing high-quality texts (Gao et al., 2022), offering feedback on text organization, language use and recommending corrections (Ohio University, 2023), logically organizing content, adding appropriate supporting details and conclusion (Fitria, 2023).”
BackgroundGao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
Research works printed but not linked: 2
- Mathur, M., & Mahapatra, S. (2022). Impact of ePortfolio assessment as an instructional strategy on students’ academic speaking skills: An experimental study. CALL-EJ, 23(3), 1–23.
- Tai, A. M. Y., Meyer, M., Varidel, M., Prodan, A., Vogel, M., Iorfino, F., & Krausz, R. M. (2023). Exploring the potential and limitations of ChatGPT for academic peer-reviewed writing: Addressing linguistic injustice and ethical concerns. Journal of Academic Language and Learning, 17(1), T16–T30.
Cited by
Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025) · Computers & Education · doi:10.1016/j.compedu.2024.105224 · Cited by 531
Wang, J., & Fan, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04787-y · Cited by 322
RetractedWang, H., Dang, A., Wu, Z., & Mac, S. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100326 · Cited by 272
Design and assessment of AI-based learning tools in higher education: a systematic review
Luo, J., Zheng, C., Yin, J., & Teo, H. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00540-2 · Cited by 333
Generative AI (GenAI) in the language classroom: A systematic review
Lee, S., Choe, H., Zou, D., & Jeon, J. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2025.2498537 · Cited by 158
Metadata
- Authors
- Mahapatra, S.
- Affiliations
- Birla Institute of Technology and Science - Hyderabad Campus (IN)
- Corresponding author
- Mahapatra, Santosh
- ORCID
- Mahapatra, Santosh
- Venue
- Smart Learning Environments
- Published
- 2024-02-13
- Volume
- 11
- Issue
- 1
- ISSN
- 2196-7091 (ISSN-L)
- OpenAlex
- W4391780419
- Semantic Scholar
- 7b9c6b77170c1dbfbc869f6ee60c60a6f33a5c8a
- Cited by
- 398
- References
- 51
- Citations by year
- 2026: 192 · 2025: 154 · 2024: 52
- FWCI
- 623.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Smart Learning Environments (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
- Social Sciences › Education › Artificial Intelligence in Education
- Social Sciences › Education › English Language Learning and Teaching
- Keywords
- ChatGPT, academic writing skills, ESL learners, formative feedback, generative AI, automated writing evaluation, dialogic feedback, student perceptions, writing instruction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Impact of ChatGPT on ESL students’ academic writing skills: a mixed methods intervention study. https://app.cochat.ai/papers/10.1186/s40561-024-00295-9
- The paper
- Mahapatra, S. (2024). Impact of ChatGPT on ESL students’ academic writing skills: a mixed methods intervention study. Smart Learning Environments. https://doi.org/10.1186/s40561-024-00295-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
