The DOI matches its Crossref record, with a small difference in title or year; 61 of the 61 references it cites were found in a registry or an index.
Comparing the effects of ChatGPT and automated writing evaluation on students’ writing and ideal L2 writing self
Open accessAbstract
The affordances of ChatGPT in language learning and teaching have gained increasing traction. While studies began to investigate the potential of ChatGPT as a feedback provider, little attention was given to ChatGPT’s potential impact on students’ writing performance and the ideal L2 writing self vis-à-vis the established automated writing evaluation systems (AWE). To address these gaps, a sequential explanatory mixed methods design was adopted. One hundred and fifty second-year university students from three writing classes in a Chinese public university were recruited and randomly divided into a ChatGPT group, an AWE group, and a control group. After an eleven-week intervention, the ANCOVA results showed that while the ChatGPT group scored significantly higher than the AWE group and the control group in post-writing performance as measured by their writing score, in terms of students’ ideal L2 writing self, the ChatGPT group performed significantly lower than the AWE group with a medium effect size. Qualitative analysis of students’ reflection papers revealed students’ (over)reliance on the tool and the accompanying loss of creativity and agency. Pedagogical implications as well as directions for future research are also discussed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“SBERT sentence embeddings can measure semantic distance between initial and revised drafts, helping identify meaning preservation, moderate extension, and obvious deviation [7].”
From Semantic Response Mechanisms of L2 Writing Revision Behavior under Generative AI Feedback (2026) · cites this paper as [7]
Citations
Cites
(1978) · Medical Entomology and Zoology · Cited by 53,488
(2017) · Cited by 16,319
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 1×
“Feedback plays an important role in enhancing learners’ writing ability (Hattie & Timperley, 2007; Hyland & Hyland, 2019).”
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1×
“These findings consolidated the concern expressed in Kasneci et al. (2023) that the effortlessly generated information could negatively impact students’ critical thinking and problem-solving skills as ChatGPT simplified the process of acquiring information, and by doing so increased students’…”
Markus, H., & Nurius, P. (1986) · American Psychologist · doi:10.1037/0003-066x.41.9.954 · Cited by 5,806
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
Perreault, K. (2010) · Manual Therapy · doi:10.1016/j.math.2010.09.003 · Cited by 4,952
2. The L2 Motivational Self System
Dörnyei, Z. (2009) · Motivation, Language Identity and the L2 Self · doi:10.21832/9781847691293-003 · Cited by 2,052
ChatGPT for Language Teaching and Learning
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023) · RELC Journal · doi:10.1177/00336882231162868 · Cited by 1,371 · cited 2×
“For ChatGPT (OpenAI, 2023), we provided students with the following exemplar prompts during the training session based on Kohnke et al. (2023) and Guo and Wang (2023) and encourage students to use all of them for every revision task: (1)You are an expert writer, please point out and correct all the…”
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113 · cited 1×
“Meanwhile, prompt engineering has gradually been regarded as an important skill that students need to be equipped with (Miao & Holmes, 2023; Tseng & Warschauer, 2023; Walter, 2024).”
Guidance for generative AI in education and research
(2023) · doi:10.54675/ewzm9535 · Cited by 808
Using ChatGPT for second language writing: pitfalls and potentials
Barrot, J. S. (2023) · Assessing Writing · doi:10.1016/j.asw.2023.100745 · Cited by 770 · cited 3×
“The studies of feedback provided by ChatGPT have followed a similar research route on examining the accuracy of ChatGPT feedback as well as its effects (Barrot, 2023a; Su et al., 2023; Wang et al., 2024).”
Collaborating with ChatGPT in argumentative writing classrooms
Su, Y., Lin, Y., & Lai, C. (2023) · Assessing Writing · doi:10.1016/j.asw.2023.100752 · Cited by 760
Song, C., & Song, Y. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1260843 · Cited by 717
The Psychology of the Language Learner
Dörnyei, Z. (2014) · doi:10.4324/9781410613349 · Cited by 700
AI-generated feedback on writing: insights into efficacy and ENL student preference
Escalante, J., Pack, A., & Barrett, A. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00425-2 · Cited by 589
Impact of AI assistance on student agency
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104967 · Cited by 548 · cited 1×
“…simplified the process of acquiring information, and by doing so increased students’ laziness and counteract their interest in independent learning, and that without guidance, students tend to rely on instead of learning from AI assistance, which weakens student agency (Darvishi et al., 2024).”
Student engagement with teacher and automated feedback on L2 writing
Zhang, Z. (., & Hyland, K. (2018) · Assessing Writing · doi:10.1016/j.asw.2018.02.004 · Cited by 546
Comparing the quality of human and ChatGPT feedback of students’ writing
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101894 · Cited by 509 · cited 3×
“Both AWE feedback and ChatGPT feedback were compared to human feedback, and both have their strengths and weaknesses (Guo & Wang, 2023; Lu et al., 2024; Steiss et al., 2024; Wang et al., 2024; Zhang & Hyland, 2018).”
Guo, K., & Wang, D. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12146-0 · Cited by 449
Automated writing evaluation: Defining the classroom research agenda
Warschauer, M., & Ware, P. (2006) · Language Teaching Research · doi:10.1191/1362168806lr190oa · Cited by 337
Wilson, J., & Czik, A. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.05.004 · Cited by 297
Partnering with AI: Intelligent writing assistance and instructed language learning
Godwin-Jones, R. (2022) · Language Learning & Technology · doi:10.64152/10125/73474 · Cited by 285
The effects of computer-generated feedback on the quality of writing
Stevenson, M., & Phakiti, A. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.11.007 · Cited by 275
Karataş, F., Abedi, F. Y., Ozek Gunyel, F., Karadeniz, D., & Kuzgun, Y. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12574-6 · Cited by 252
Barrot, J. S. (2023) · Computer Assisted Language Learning · doi:10.1080/09588221.2021.1936071 · Cited by 248 · cited 2×
“…such as Grammarly, Criterion, and Pigai, were designed to provide both summative scores and formative feedback for learners’ writing, based on natural language processing, artificial intelligence, latent semantic analysis, etc. (Barrot, 2021; Shi & Aryadoust, 2023, 2024; Warschauer & Ware, 2006).”
Future selves, enjoyment and anxiety as predictors of L2 writing achievement
Tahmouresi, S., & Papi, M. (2021) · Journal of Second Language Writing · doi:10.1016/j.jslw.2021.100837 · Cited by 236
A systematic review of AI-based automated written feedback research
Shi, H., & Aryadoust, V. (2024) · ReCALL · doi:10.1017/s0958344023000265 · Cited by 222
Self-citationTeacher written corrective feedback: Less is more
Lee, I. (2019) · Language Teaching · doi:10.1017/s0261444819000247 · Cited by 221
Is Chatgpt a menace for creative writing ability? An experiment
Niloy, A. C., Akter, S., Sultana, N., Sultana, J., & Rahman, S. I. U. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12929 · Cited by 198
Motivational Mechanisms Underlying Second Language Achievement: A Regulatory Focus Perspective
Papi, M., & Khajavy, G. H. (2021) · Language Learning · doi:10.1111/lang.12443 · Cited by 191
The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis
Zhai, N., & Ma, X. (2022) · Journal of Educational Computing Research · doi:10.1177/07356331221127300 · Cited by 186
Boudouaia, A., Mouas, S., & Kouider, B. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241247465 · Cited by 186
Can ChatGPT effectively complement teacher assessment of undergraduate students’ academic writing?
Lu, Q., Yao, Y., Xiao, L., Yuan, M., Wang, J., & Zhu, X. (2024) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2024.2301722 · Cited by 173
ChatGPT's capabilities in providing feedback on undergraduate students’ argumentation: A case study
Wang, L., Chen, X., Wang, C., Xu, L., Shadiev, R., & Li, Y. (2023) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2023.101440 · Cited by 153
Artificial Intelligence in Foreign Language Learning and Teaching
Schmidt, T. D., & Strassner, T. (2022) · Anglistik · doi:10.33675/angl/2022/1/14 · Cited by 151
The impact of ChatGPT on L2 writing and expected responses: Voice from doctoral students
Zou, M., & Huang, L. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12397-x · Cited by 139
Contexts and Issues in Feedback on L2 Writing
Hyland, K., & Hyland, F. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108635547.003 · Cited by 137
Kohnke, L. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100279 · Cited by 132
ChatGPT as a Language Learning Tool: An Emerging Technology Report
Barrot, J. S. (2023) · Technology Knowledge and Learning · doi:10.1007/s10758-023-09711-4 · Cited by 130
Ding, L., & Zou, D. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12402-3 · Cited by 122
AI-writing tools in education: If you can’t beat them, join them
Tseng, W., & Warschauer, M. (2023) · Journal of China Computer-Assisted Language Learning · doi:10.1515/jccall-2023-0008 · Cited by 119 · cited 3×
“Compared to AWE, ChatGPT has far more affordances (Barrot, 2023a, 2023b; Godwin-Jones, 2024; Tseng & Warschauer, 2023), and is not pre-customized for learners of certain language proficiency.”
Kim, T.-Y. (2009) · Multilingual Matters eBooks · doi:10.21832/9781847691293-015 · Cited by 115
A systematic review of automated writing evaluation systems
Shi, H., & Aryadoust, V. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11200-7 · Cited by 111
Self-citationAutomated Grammatical Error Detection for Language Learners
Leacock, C., Chodorow, M. S., Gamon, M., & Tetreault, J. (2010) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-02137-4 · Cited by 105
Exploring the potential of ChatGPT in assessing L2 writing accuracy for research purposes
Pfau, A., Polio, C. G., & Xu, Y. (2023) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2023.100083 · Cited by 91
Zheng, C., Wang, L., & Chai, C. S. (2021) · Computer Assisted Language Learning · doi:10.1080/09588221.2021.1946562 · Cited by 87
Self-citationExploring ChatGPT as a writing assessment tool
Bucol, J. L., & Sangkawong, N. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2024.2363901 · Cited by 79
The effects of generative AI usage in EFL classrooms on the l2 motivational self system
Huang, J., & Mizumoto, A. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13071-6 · Cited by 73
Zhao, P., Zhu, X., Yao, Y., & Liao, X. (2023) · System · doi:10.1016/j.system.2023.103033 · Cited by 65
Waer, H. (2021) · Innovation in Language Learning and Teaching · doi:10.1080/17501229.2021.1914062 · Cited by 60
Xu, J., & Zhang, S. (2021) · The Asia-Pacific Education Researcher · doi:10.1007/s40299-021-00577-7 · Cited by 58
Computer-Mediated Corrective Feedback to Improve L2 Writing Skills: A Meta-Analysis
Mohsen, M. A. (2022) · Journal of Educational Computing Research · doi:10.1177/07356331211064066 · Cited by 56
Automated writing evaluation use in second language classrooms: A research synthesis
Karatay, Y., & Karatay, L. (2024) · System · doi:10.1016/j.system.2024.103332 · Cited by 49
Mujtaba, S. M., Reynolds, B. L., Parkash, R., & Singh, M. K. M. (2021) · Assessing Writing · doi:10.1016/j.asw.2021.100566 · Cited by 45
Automated written corrective feedback: Error-correction performance and timing of delivery
Ranalli, J., & Yamashita, T. (2022) · Language learning & technology · doi:10.64152/10125/73465 · Cited by 38
Utilizing OpenAI 's GPT ‐4 for written feedback
Carlson, M., Pack, A., & Escalante, J. (2023) · TESOL Journal · doi:10.1002/tesj.759 · Cited by 29
The Bloomsbury Handbook of Language Learning and Technology
Boulton, A., & Pérez-Paredes, P. (2024) · Bloomsbury Publishing Plc eBooks · doi:10.5040/9781350340350 · Cited by 24
Aubrey, S., & Philpott, A. (2019) · RELC Journal · doi:10.1177/0033688219865409 · Cited by 16
Self-citationExpanding the learning ecology and autonomy of language learners with mobile technologies
Stockwell, G., & Wang, Y. (2024) · DOAJ (DOAJ: Directory of Open Access Journals) · doi:10.30191/ets.202404_27(2).sp05 · Cited by 13
Assessing reading comprehension in CET-4: a cross-sectional case study
Huang, C. (2022) · Interactive Learning Environments · doi:10.1080/10494820.2022.2061012 · Cited by 5
Research works printed but not linked: 5
- Burstein J. Chodorow M. & Leacock C. (2003). Criterion online essay evaluation: An application for automated evaluation of student essays [Paper presentation]. Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence Acapulco: Mexico.
- 10.64152/10125/73570 · DOI
- Saldaña, J. (2016). The coding manual for qualitative researchers (3rd ed.). Sage Publications.
- Shi, J., & Shuai, P. (2010). English writing: From sentence to paragraph to theme. Foreign Language Teaching and Research Press.
- Stockwell, G., & Wang, Y. (2024b). Motivation and autonomy. In R. Hampel & U. Stickler (Eds.), The Bloomsbury Handbook of Language Learning and Technology (pp. 144–156). Bloomsbury Academic.
Cited by
Sorted by citations
Generative AI and L2 Written Feedback Studies: A Scoping Review
Crosthwaite, P., & Sun, S. (2026) · RELC Journal · doi:10.1177/00336882251386530 · Cited by 50
Ding, L., Zou, D., & Kohnke, L. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13775-3 · Cited by 59
Pawlak, M., Kruk, M., Shirvan, M. E., Taherian, T., & Karimpour, S. (2025) · Computer Assisted Language Learning · doi:10.1080/09588221.2025.2498094 · Cited by 29
Generative AI vs. Teachers: Feedback Quality, Feedback Uptake, and Revision
Yu, H., & Xie, Q. (2025) · Language Teaching Research Quarterly · doi:10.32038/ltrq.2025.47.07 · Cited by 22
Teng, L. S., Deng, X., & Yang, J. (2026) · System · doi:10.1016/j.system.2026.103970 · Cited by 16
Metadata
- Authors
- Shi, H., Chai, C. S., Zhou, S., & Aubrey, S.
- Affiliations
- Chinese University of Hong Kong (HK)
- Corresponding author
- Shi, Huawei
- Venue
- Computer Assisted Language Learning
- Publisher
- Informa UK Limited
- Published
- 2025-02-03
- Volume
- 39
- Issue
- 4
- Pages
- 1148-1175
- ISSN
- 0958-8221 (ISSN-L)
- 1744-3210
- OpenAlex
- W4407108474
- Cited by
- 109
- References
- 61
- Citations by year
- 2026: 87 · 2025: 22
- FWCI
- 387.04
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Computer Assisted Language Learning (publishedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
- Social Sciences › Education › English Language Learning and Teaching
- Social Sciences › Psychology › Developmental and Educational Psychology › Second Language Acquisition and Learning
- Keywords
- ChatGPT, automated writing evaluation, second language writing, writing performance, AI-generated feedback, student creativity, student agency, university students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
