Enhancing university level English proficiency with generative AI: empirical insights into automated feedback and learning outcomes
Open accessAbstract
This paper investigates the effects of large language model (LLM) based feedback on the essay writing proficiency of university students in Hong Kong. It focuses on exploring the potential improvements that generative artificial intelligence (AI) can bring to student essay revisions, its effect on student engagement with writing tasks, and the emotions students experience while undergoing the process of revising written work. Utilizing a randomized controlled trial, it draws comparisons between the experiences and performance of 918 language students at a Hong Kong university, some of whom received generated feedback (GPT-3.5-turbo LLM) and some of whom did not. The impact of AI-generated feedback is assessed not only through quantifiable metrics, entailing statistical analysis of the impact of AI feedback on essay grading, but also through subjective indices, student surveys that captured motivational levels and emotional states, as well as thematic analysis of interviews with participating students. The incorporation of AI-generated feedback into the revision process demonstrated significant improvements in the caliber of students’ essays. The quantitative data suggests notable effect sizes of statistical significance, while qualitative feedback from students highlights increases in engagement and motivation as well as a mixed emotional experience during revision among those who received AI feedback.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · see citing works
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · see citing works
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242 · see citing works
Thematic networks: an analytic tool for qualitative research
Attride-Stirling, J. (2001) · Qualitative Research · doi:10.1177/146879410100100307 · Cited by 5,506 · see citing works
Artificial Intelligence in Education: A Review
Chen, L., Chen, P., & Lin, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2988510 · Cited by 4,133 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Eccles, J. S., & Wigfield, A. (2020) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2020.101859 · Cited by 2,368 · see citing works
Vision, challenges, roles and research issues of Artificial Intelligence in Education
Hwang, G.-J., Xie, H., Wah, B. W., & Gašević, D. (2020) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2020.100001 · Cited by 1,435 · see citing works
Darwin Holmes, A. G. (2020) · Shanlax International Journal of Education · doi:10.34293/education.v8i4.3232 · Cited by 1,341 · see citing works
Evaluation of unsupervised semantic mapping of natural language with Leximancer concept mapping
Smith, A. E., & Humphreys, M. S. (2006) · Behavior Research Methods · doi:10.3758/bf03192778 · Cited by 1,206 · see citing works
AI-enabled adaptive learning systems: A systematic mapping of the literature
Kabudi, T., Pappas, I., & Olsen, D. H. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100017 · Cited by 734 · see citing works
Huang, A. Y. Q., Lu, O. H. T., & Yang, S. J. H. (2023) · Computers & Education · doi:10.1016/j.compedu.2022.104684 · Cited by 605 · see citing works
ChatGPT: A revolutionary tool for teaching and learning mathematics
Wardat, Y., Tashtoush, M. A., AlAli, R. M., & Jarrah, A. M. (2023) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste/13272 · Cited by 593 · see citing works
An automated essay scoring systems: a systematic literature review
Ramesh, D., & Sanampudi, S. K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10068-2 · Cited by 550 · see citing works
Comparing the quality of human and ChatGPT feedback of students’ writing
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101894 · Cited by 509 · see citing works
Madigan, D. J., & Kim, L. E. (2021) · International Journal of Educational Research · doi:10.1016/j.ijer.2020.101714 · Cited by 481 · see citing works
Magaldi, D., & Berler, M. (2020) · Encyclopedia of Personality and Individual Differences · doi:10.1007/978-3-319-24612-3_857 · Cited by 433 · see citing works
Lee, D., Arnold, M., Srivastava, A., Plastow, K., Strelan, P., Ploeckl, F., Lekkas, D., & Palmer, E. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100221 · Cited by 402 · see citing works
Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT
Dai, W., Lin, J., Jin, H., Li, T., Tsai, Y.-S., Gašević, D., & Chen, G. (2023) · 2023 IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt58122.2023.00100 · Cited by 395 · see citing works
A review of feedback models and typologies: Towards an integrative model of feedback elements
Panadero, E., & Lipnevich, A. A. (2022) · Educational Research Review · doi:10.1016/j.edurev.2021.100416 · Cited by 389 · see citing works
AI literacy and its implications for prompt engineering strategies
Knoth, N., Tolzin, A., Janson, A., & Leimeister, J. M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100225 · Cited by 368 · see citing works
Formative Assessment and Writing: A Meta-Analysis
Graham, S., Hebert, M., & Harris, K. R. (2015) · The Elementary School Journal · doi:10.1086/681947 · Cited by 340 · see citing works
Meyer, J., Jansen, T., Schiller, R., Liebenow, L. W., Steinbach, M., Horbach, A., & Fleckenstein, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100199 · Cited by 311 · see citing works
Towards artificial intelligence-based assessment systems
Luckin, R. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0028 · Cited by 303 · see citing works
Essel, H. B., Vlachopoulos, D., Essuman, A. B., & Amankwa, J. O. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100198 · Cited by 296 · see citing works
Basic Research Methods: An Entry to Social Science Research
Guthrie, G. (2010) · SAGE Publications India Pvt Ltd eBooks · doi:10.4135/9788132105961 · Cited by 194 · see citing works
Applying large language models and chain-of-thought for automatic scoring
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100213 · Cited by 186 · see citing works
Role of activity-based learning and ChatGPT on students' performance in education
Al Shloul, T., Mazhar, T., Abbas, Q., Iqbal, M., Ghadi, Y. Y., Shahzad, T., Mallek, F., & Hamam, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100219 · Cited by 173 · see citing works
Al-khresheh, M. H. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100218 · Cited by 163 · see citing works
Automated feedback and writing: a multi-level meta-analysis of effects on students' performance
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1162454 · Cited by 145 · see citing works
Interviews in qualitative research
Peters, K., & Halcomb, E. (2015) · Nurse Researcher · doi:10.7748/nr.22.4.6.s2 · Cited by 141 · see citing works
Eight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137 · see citing works
Automatic assessment of text-based responses in post-secondary education: A systematic review
Gao, R., Merzdorf, H. E., Anwar, S., Hipwell, M. C., & Srinivasa, A. R. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100206 · Cited by 137 · see citing works
Natural Language Generation Using Deep Learning to Support MOOC Learners
Li, C., & Xing, W. (2021) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-020-00235-x · Cited by 133 · see citing works
Mapping Artificial Intelligence in Education Research: a Network-based Keyword Analysis
Feng, S., & Law, N. W. Y. (2021) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00244-4 · Cited by 116 · see citing works
The future of feedback: Motivating performance improvement through future-focused feedback
Gnepp, J., Klayman, J., Williamson, I. O., & Barlas, S. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234444 · Cited by 112 · see citing works
Hahn, M. G., Navarro, S. M. B., De La Fuente Valentin, L., & Burgos, D. (2021) · IEEE Access · doi:10.1109/access.2021.3100890 · Cited by 111 · see citing works
Mertens, U., Finn, B., & Lindner, M. A. (2022) · Journal of Educational Psychology · doi:10.1037/edu0000764 · Cited by 110 · see citing works
Motivating revision of drafts through formative feedback
McGarrell, H., & Verbeem, J. (2007) · ELT Journal · doi:10.1093/elt/ccm030 · Cited by 102 · see citing works
Bressane, A., Zwirn, D., Essiptchouk, A. M., Saraiva, A. C. V., Carvalho, F., Formiga, J. K. S., Medeiros, L. C. D. C., & Negri, R. G. (2023) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2023.100196 · Cited by 96 · see citing works
Should I grade or should I comment: Links among feedback, emotions, and performance
Lipnevich, A. A., Murano, D., Krannich, M., & Goetz, T. (2021) · Learning and Individual Differences · doi:10.1016/j.lindif.2021.102020 · Cited by 69 · see citing works
Foundation Models for Decision Making: Problems, Methods, and Opportunities
Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., & Schuurmans, D. (2023) · arXiv · doi:10.48550/arxiv.2303.04129 · Cited by 60 · see citing works
Jacobsen, L. J., & Weber, K. E. (2023) · doi:10.31219/osf.io/cr257 · Cited by 57 · see citing works
Aslan, S., Durham, L., Alyüz, N., Okur, E., Sharma, S., Savur, C., & Nachman, L. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100220 · Cited by 52 · see citing works
Alvero, A., Arthurs, N., antonio, A. L., Domingue, B. W., Gebre-Medhin, B., Giebel, S., & Stevens, M. L. (2020) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3375627.3375871 · Cited by 50 · see citing works
Crossley, S. A., Baffour, P., Tian, Y., Picou, A., Benner, M., & Boser, U. (2022) · Assessing Writing · doi:10.1016/j.asw.2022.100667 · Cited by 47 · see citing works
An Integrative Framework for Artificial Intelligence Education
Pat Langley, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019670 · Cited by 44 · see citing works
Improving EFL students’ cultural awareness: Reframing moral dilemmatic stories with ChatGPT
Zheng, Y., & Stewart, N. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100223 · Cited by 44 · see citing works
Using data mining techniques to automatically construct concept maps for adaptive learning systems
Chen, S., & Bai, S.-M. (2009) · Expert Systems with Applications · doi:10.1016/j.eswa.2009.12.060 · Cited by 42 · see citing works
INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
Chia, Y. K., Hong, P., Bing, L., & Poria, S. (2023) · arXiv · doi:10.48550/arxiv.2306.04757 · Cited by 36 · see citing works
Augmenting assessment with AI coding of online student discourse: A question of reliability
Misiejuk, K., Kaliisa, R., & Scianna, J. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100216 · Cited by 33 · see citing works
Schrader, C., & Kalyuga, S. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.102374 · Cited by 31 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- McCormick, K. (2015). SPSS statistics for dummies. John Wiley.
- Salcedo, J., & McCormick, K. (2020). SPSS statistics (4th ed.). John Wiley.
- Chan, Sumie Tsz Sum, Noble Po Kan Lo, and Alan Man Him Wong. "Enhancing university level English proficiency with generative AI: Empirical insights into automated feedback and learning outcomes". Contemporary Educational Technology 2024 16 no. 4 (2024): ep541. https://doi.org/10.30935/cedtech/15607 Chan, Sumie Tsz Sum et al. "Enhancing university level English proficiency with generative AI: Empirical insights into automated feedback and learning outcomes". Contemporary Educational Technology, vol. 16, no. 4, 2024, ep541. https://doi.org/10.30935/cedtech/15607 Chan STS, Lo NPK, Wong AMH. Enhancing university level English proficiency with generative AI: Empirical insights into automated feedback and learning outcomes. CONT ED TECHNOLOGY. 2024;16(4):ep541. https://doi.org/10.30935/cedtech/15607
Cited by
Sorted by citations
Generative AI and L2 Written Feedback Studies: A Scoping Review
Crosthwaite, P., & Sun, S. (2026) · RELC Journal · doi:10.1177/00336882251386530 · Cited by 50 · see citing works
Alshehri, O. A. O., Lamouchi, A., Abdellatif, M. S., Al-Dosari, M. N. A., Mekheimer, M., & Nemt-allah, M. A. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1830103 · Cited by 2 · see citing works
“…visible and deliberate—students compare AI-generated alternatives against their own drafts, evaluate competing formulations, and make explicit accept-or-reject decisions that strengthen metacognitive awareness of compositional choices (Chan et al., 2024; Hong et al., 2025; Nguyen et al., 2024).”
- Authors
- Chan, S. T. S., Lo, N. P. K., & Wong, A. M. H.
- Affiliations
- Chinese University of Hong Kong (HK)
- Hong Kong Polytechnic University (HK)
- Corresponding author
- Lo, Noble Po Kan
- Venue
- Contemporary Educational Technology
- Publisher
- Bastas Publications
- Published
- 2024-11-13
- Volume
- 16
- Issue
- 4
- Pages
- ep541
- ISSN
- 1309-517X (ISSN-L)
- OpenAlex
- W4404332403
- Cited by
- 56
- References
- 52
- Citations by year
- 2026: 35 · 2025: 21
- FWCI
- 21.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Contemporary Educational Technology (publishedVersion, cc-by)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (publishedVersion, cc-by)
- Lancaster EPrints (Lancaster University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
- Social Sciences › Education › Student Assessment and Feedback
- Keywords
- generative AI, large language models, AI-generated feedback, essay writing, English proficiency, university students, essay revision, student motivation, student engagement, emotions in writing, GPT-3.5-turbo, writing instruction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review