From process to product: writing engagement and performance of EFL learners under computer-generated feedback instruction
Open accessThe DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) technology is gradually penetrating the domain of education, opening up many possibilities for teaching and learning. Many educators, faced with the burden of commenting on substantial student essays, have introduced automated writing evaluation (AWE) into second language (L2) writing considering its affordance of immediate scores and diagnostic information. However, students’ processing strategies and perceptions of such computer-generated feedback and its impact on student writing quality, particularly as mediated by language proficiency, remain under-explored. This study examines the impact of Pigai , a Chinese AWE system, on revision processes and writing products of 42 English as a foreign language (EFL) learners with varying language levels by analyzing feedback points, feedback uptake, text quality in complexity, accuracy, and fluency (CAF), and perceptions. The findings are as follows. When confronted with AWE instruction, the majority of student work focuses on correcting errors, but higher-level students exhibit an emphasis on language improvement beyond the surface level compared to lower-level students. According to CAF measures, automated feedback exerts greater effects on accuracy for unskilled learners and lexical complexity for skilled learners in the development of interlanguage. Learner profiles and perceptions of students at different levels are explored along four dimensions: writing quality, cognitive engagement, behavioral engagement, and affective engagement. Finally, the potential issues of such technology-based writing instruction are pointed out.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, AI tools provide immediate, individualized diagnostic feedback on linguistic accuracy, syntactic complexity, and genre-level organization—otherwise constrained by class size and instructor availability—enabling students to identify and repair surface and structural weaknesses across multiple drafting cycles (Chan et al., 2024; Mekheimer, 2025; Shen et al., 2023; Song and Song, 2023).”
From From tool to scaffold: structured human–AI collaboration and its effects on academic writing and digital critical thinking among Saudi EFL learners (2026) · cites this paper as Shen et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Few studies have looked at the long-term effects of ChatGPT and similar tools on the development of writing skills, particularly in actual classroom settings.”
From Evaluating the impact of using ChatGPT on EFL students’ argumentative skills (2026) · cites this paper as Chen et al. [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Chen et al. [5] found that computer-generated feedback could enhance EFL learners’ engagement and improve the quality of written products”
From Evaluating the impact of using ChatGPT on EFL students’ argumentative skills (2026) · cites this paper as Chen et al. [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some studies found stronger effects among lower-proficiency learners (Wei et al., 2023; Shen et al., 2023)”
From The Use of Artificial Intelligence Tools in English Academic Writing among University Students: A Scoping Review (2025) · cites this paper as Shen et al., 2023
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Qualitative Inquiry and Research Design: Choosing Among Five Approaches
Lewis, S. (2015) · Health Promotion Practice · doi:10.1177/1524839915580941 · Cited by 22,105
Cohen, L., Manion, L., & Morrison, K. (2007) · Routledge eBooks · doi:10.4324/9780203029053 · Cited by 20,105
(2017) · Cited by 16,319
Complex Systems and Applied Linguistics
Cameron, L., & Larsen-Freeman, D. (2007) · International Journal of Applied Linguistics · doi:10.1111/j.1473-4192.2007.00148.x · Cited by 1,317
Error feedback in L2 writing classes
Ferris, D., & Roberts, B. (2001) · Journal of Second Language Writing · doi:10.1016/s1060-3743(01)00039-x · Cited by 981
Complexity, accuracy and fluency in second language acquisition
Housen, A., & Kuiken, F. (2009) · Applied Linguistics · doi:10.1093/applin/amp048 · Cited by 863 · cited 1×
“It might evolve slowly since it is more dependent on the psycholinguistic process when applying L2 knowledge (Housen and Kuiken, 2009).”
BackgroundLarsen-Freeman, D. (2006) · Applied Linguistics · doi:10.1093/applin/aml029 · Cited by 805
LU, X. (2011) · TESOL Quarterly · doi:10.5054/tq.2011.240859 · Cited by 744
The Relationship of Lexical Richness to the Quality of ESL Learners’ Oral Narratives
LU, X. (2012) · The Modern Language Journal · doi:10.1111/j.1540-4781.2011.01232_1.x · Cited by 526
(2013) · Studies in corpus linguistics · doi:10.1075/scl · Cited by 467
Koltovskaia, S. (2020) · Assessing Writing · doi:10.1016/j.asw.2020.100450 · Cited by 447 · cited 1×
“Most scholars assessed AWE systems around how students engaged in the revision process primarily by qualitative instruments, such as interviews, self-reports, verbal protocols, and observations (Koltovskaia, 2020; Lee, 2020; Zhang, 2020; Fu and Liu, 2022).”
BackgroundExploring learner engagement with written corrective feedback in a Chinese tertiary EFL classroom
Han, Y., & Hyland, F. (2015) · Journal of Second Language Writing · doi:10.1016/j.jslw.2015.08.002 · Cited by 428
LEARNERS’ PROCESSING, UPTAKE, AND RETENTION OF CORRECTIVE FEEDBACK ON WRITING: Case Studies
Storch, N., & Wigglesworth, G. (2010) · Studies in Second Language Acquisition · doi:10.1017/s0272263109990532 · Cited by 398
THE EFFECTS OF PLANNING ON FLUENCY, COMPLEXITY, AND ACCURACY IN SECOND LANGUAGE NARRATIVE WRITING
Ellis, R., & Yuan, F. (2004) · Studies in Second Language Acquisition · doi:10.1017/s0272263104026130 · Cited by 368
Automated writing evaluation: Defining the classroom research agenda
Warschauer, M., & Ware, P. (2006) · Language Teaching Research · doi:10.1191/1362168806lr190oa · Cited by 337
The Linguistic Development of Students of English as a Second Language in Two Written Genres
Yoon, H.-J., & Polio, C. (2017) · TESOL Quarterly · doi:10.1002/tesq.296 · Cited by 317
Chen, C.-F. E., & Cheng, W.-Y. E. C. (2008) · Language Learning & Technology · doi:10.64152/10125/44145 · Cited by 304
Predicting patterns of grammatical complexity across language exam task types and proficiency levels
Biber, D., Gray, B., & Staples, S. (2016) · Applied Linguistics · doi:10.1093/applin/amu059 · Cited by 297
Wilson, J., & Czik, A. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.05.004 · Cited by 297
Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction
Li, J., Link, S., & Hegelheimer, V. (2015) · Journal of Second Language Writing · doi:10.1016/j.jslw.2014.10.004 · Cited by 282
The effects of computer-generated feedback on the quality of writing
Stevenson, M., & Phakiti, A. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.11.007 · Cited by 275 · cited 1×
“Holistic score and writing accuracy are the most popular measures in the AWE literature, but the results remain inconclusive (Stevenson and Phakiti, 2014).”
BackgroundLink, S., Mehrzad, M., & Rahimi, M. (2022) · Computer Assisted Language Learning · doi:10.1080/09588221.2020.1743323 · Cited by 275
Zhang, Z. (. (2020) · Assessing Writing · doi:10.1016/j.asw.2019.100439 · Cited by 256 · cited 4×
“It may be inferred from the variation in clausal use that high-level learners who receive AWE feedback are more likely to elicit metacognitive and cognitive operations in the processing stage, which aligns with Zhang (2020).”
BackgroundInfluentialFostering student engagement with feedback: An integrated approach
Zhang, Z. (., & Hyland, K. (2022) · Assessing Writing · doi:10.1016/j.asw.2021.100586 · Cited by 235
Automated written corrective feedback: how well can students make use of it?
Ranalli, J. (2018) · Computer Assisted Language Learning · doi:10.1080/09588221.2018.1428994 · Cited by 228 · cited 1×
“First of all, an overwhelming majority of studies have merely sampled learners at a single proficiency level (Ranalli, 2018).”
BackgroundAn investigation into current measures of linguistic accuracy in second language writing research
Polio, C., & Shea, M. C. (2014) · Journal of Second Language Writing · doi:10.1016/j.jslw.2014.09.003 · Cited by 185
Wang, Y.-J., Shang, H.-F., & Briody, P. (2013) · Computer Assisted Language Learning · doi:10.1080/09588221.2012.655300 · Cited by 185 · cited 2×
“Items on writing quality were adapted from Wang et al. (2013), and items on cognitive engagement, behavioral engagement, and affective engagement were adapted from Han and Hyland (2015).”
BackgroundMethodRevising in two languages: A multi-dimensional comparison of online writing revisions in L1 and FL
Stevenson, M., Schoonen, R., & de Glopper, K. (2006) · Journal of Second Language Writing · doi:10.1016/j.jslw.2006.06.002 · Cited by 176
In the face of fallible AWE feedback: how do students respond?
Bai, L., & Hu, G. (2017) · Educational Psychology · doi:10.1080/01443410.2016.1223275 · Cited by 168
Validity arguments for diagnostic assessment using automated writing evaluation
Chapelle, C. A., Cotos, E., & Lee, J. (2015) · Language Testing · doi:10.1177/0265532214565386 · Cited by 156
The role of automated writing evaluation holistic scores in the ESL classroom
Li, Z., Link, S., Ma, H., Yang, H., & Hegelheimer, V. H. (2014) · System · doi:10.1016/j.system.2014.02.007 · Cited by 130
A corpus-based comparison of syntactic complexity in NNS and NS university students’ writing
Ai, H., & Lu, X. (2013) · Studies in Corpus Linguistics · doi:10.1075/scl.59.15ai · Cited by 126
Zhang, L. J., & Cheng, X. (2021) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2021.101043 · Cited by 122
Chodorow, M., Gamon, M., & Tetreault, J. (2010) · Language Testing · doi:10.1177/0265532210364391 · Cited by 115
The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis
Zhai, N., & Ma, X. (2023) · Journal of Educational Computing Research · doi:10.1177/07356331221127300 · Cited by 115
Does Automated Feedback Help Students Learn to Write?
Kellogg, R. T., Whiteford, A. P., & Quinlan, T. (2010) · Journal of Educational Computing Research · doi:10.2190/ec.42.2.c · Cited by 88
The accuracy of computer-assisted feedback and students’ responses to it
Lavolette, E., Polio, C., & Kahng, J. (2015) · Language Learning & Technology · doi:10.64152/10125/44417 · Cited by 88
Maamuujav, U. (2021) · Assessing Writing · doi:10.1016/j.asw.2021.100540 · Cited by 84
Lei, L., Wen, J., & Yang, X. (2022) · Journal of Second Language Writing · doi:10.1016/j.jslw.2022.100962 · Cited by 83
Enhancing the grammatical accuracy of EFL writing by using an AWE-assisted process approach
Liao, H.-C. (2016) · System · doi:10.1016/j.system.2016.02.007 · Cited by 82 · cited 1×
“Liao (2016) assigned four writing tasks to 63 EFL learners who received guidance from Criterion to investigate their grammatical accuracy development, perceptions, and possible factors mediating acquisition.”
BackgroundA guide to doing statistics in second language research using SPSS and R
(2016) · Choice Reviews Online · doi:10.5860/choice.195411 · Cited by 82
Jiang, L., Yu, S., & Wang, C. (2020) · System · doi:10.1016/j.system.2020.102302 · Cited by 71
Xu, J., & Zhang, S. (2021) · The Asia-Pacific Education Researcher · doi:10.1007/s40299-021-00577-7 · Cited by 69
Lee, C. (2020) · Computer Assisted Language Learning · doi:10.1080/09588221.2018.1544152 · Cited by 59
Li, Z., Feng, H.-H., & Saricaoglu, A. (2017) · CALICO Journal · doi:10.1558/cj.26382 · Cited by 52
Huang, F., Sánchez‐Prieto, J. C., Teo, T. P., García‐Peñalvo, F. J., Sánchez, E. M. T., & Zhao, C. (2020) · Educational Technology Research and Development · doi:10.1007/s11423-020-09806-0 · Cited by 50
Fan, N. (2023) · Sage Open · doi:10.1177/21582440231181296 · Cited by 42
Hassanzadeh, M., & Fotoohnejad, S. (2021) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12587 · Cited by 40
Reynolds, B. L., Kao, C.-W., & Huang, Y.-Y. (2021) · The Asia-Pacific Education Researcher · doi:10.1007/s40299-021-00597-3 · Cited by 38
The impact of Automated Essay Scoring (AES) for improving English language learner's essay writing
Choi, J. (2010) · Cited by 23
James, C. (2006) · Assessing Writing · doi:10.1016/j.asw.2007.01.002 · Cited by 21
EFL Learner Engagement in Automatic Written Evaluation
Fu, H., & Liu, X. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.871707 · Cited by 21
Research works printed but not linked: 1
- Wolfe-Quintero K. Inagaki S. Kim H.-Y. (1998). Second language development in writing: Measures of fluency, accuracy, & complexity. Honolulu: Second Language Teaching & Curriculum Center, University of Hawaii at Manoa.
Cited by
Cai, L., Msafiri, M. M., & Kangwa, D. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13112-0 · Cited by 104
The Impact of AI on the Personal and Collaborative Learning Environments in Higher Education
Msambwa, M. M., Wen, Z., & Daniel, K. (2025) · European Journal of Education · doi:10.1111/ejed.12909 · Cited by 76
Daniel, K., Msambwa, M. M., & Wen, Z. (2025) · European Journal of Education · doi:10.1111/ejed.70036 · Cited by 71
A systematic review of AI-based feedback in educational settings
Durak, H. Y., & Onan, A. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-025-00428-1 · Cited by 89
Min, H., Abbasi, B. N., & Jing, H. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05817-5 · Cited by 76
Metadata
- Authors
- Shen, C., Shi, P., Guo, J., Xu, S., & Tian, J.
- Affiliations
- Chang'an University (CN)
- Xi'an Jiaotong University (CN)
- Xi'an Jiaotong University City College (CN)
- Air Force Engineering University (CN)
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2023-10-23
- Volume
- 14
- Pages
- 1258286-1258286
- ISSN
- 1664-1078 (ISSN-L)
- PMC
- PMC10701688
- OpenAlex
- W4387878129
- Cited by
- 42
- References
- 54
- Citations by year
- 2026: 17 · 2025: 21 · 2024: 4
- FWCI
- 47.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Psychology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
- Social Sciences › Psychology › Developmental and Educational Psychology › Second Language Acquisition and Learning
- Social Sciences › Education › English Language Learning and Teaching
- Keywords
- automated writing evaluation, automated feedback, EFL writing, second language writing, revision strategies, feedback uptake, language proficiency, interlanguage development, writing quality, student engagement, student perceptions, error correction, lexical complexity, fluency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
