CoChat
Sign Up
Cited by 11Artificial Intelligence in EducationStudent Assessment and FeedbackEFL/ESL Teaching and Learning

From red ink to algorithms: investigating the use of large language models in academic writing feedback

Open access
Jovic, Marina, Papakonstantinidis, Stavros, Kirkpatrick, Robert·2025·Language Testing in Asia
DOI 10.1186/s40468-025-00389-2
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper explores the emerging role of large language models (LLMs) in academic feedback for student writing. It examines feedback provided by two human graders and two LLMs, ChatGPT-4.5 and Claude-3.7 Sonnet, on argumentative and reflective essays written by students from diverse linguistic backgrounds. Our investigation addresses two research questions: how human and AI evaluators differ in assessing distinct academic essay genres and how conversational prompting and genre influence the equity and actionability of AI-generated feedback. A six-dimension evaluation rubric reveals that human graders excel at understanding context and offering holistic guidance, while AI-generated feedback demonstrates superior structure and efficiency, particularly in identifying mechanical and organizational issues. AI systems using conversational prompting significantly outperformed both human graders and zero-shot AI, with notable gains in future orientation and conciseness for argumentative essays. Both human and AI evaluators generally produced higher-quality feedback for reflective essays, with 1.3 + point advantages. While AI systems applied more consistent evaluation criteria across different student groups, they showed limitations in deeper evaluation of argument structure and personalized revision strategies. The paper demonstrates how educators can enhance AI-generated feedback through tailored conversational prompting strategies, suggesting that a blended approach combining AI’s structural consistency with human expertise in higher-order analysis offers the most effective educational outcomes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s40468-025-00389-2

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The technostress literature identifies emotional and behavioral effects of using technologies such as anxiety, overload, and avoidance behaviors.

    “Simultaneously, the technostress literature provides valuable information on emotional and behavioral effects of using technologies such as anxiety, overload, and avoidance behaviors (Kasneci et al., 2023; Jovic et al., 2025; Chan, 2023).”

    From Mapping the impact of generative AI in higher education: A scoping review of psychological and equity dimensions. (2026) · cites this paper as Jovic et al., 2025

    The AI feedback quality: identified limitations and challenges section discusses overdependence on technology and problems with AI-generated feedback, not technostress or its effects on users. Searches for technostress, anxiety, overload, and avoidance behaviors found no passage in our copy addressing the attributed claim; page numbers were unavailable. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. Alers, H., Malinowska, A., Meghoe, G., spsampsps Apfel, E. (2024). Using ChatGPT-4 to grade open question exams. In Lecture Notes in Networks and Systems: Vol. 919 LNNS. https:// doi. org/ 10. 1007/ 978-3-031-53960-2_1 · DOI
  2. Banihashem, S. K., Kerman, N. T., Noroozi, O., Moon, J., & Drachsler, H. (2024). Feedback sources in essay writing: Peer-gen- erated or AI-generated feedback? International Journal of Educational Technology in Higher Education,21(1), Article 23.
  3. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) On the dangers of stochastic parrots: Can language models be too big?. In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610- 623). https:// doi. org/ 10. 1145/ 34421 88. 344592 · DOI
  4. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice,5(1), 7-74.
  5. Bouziane, K. (2024). AI versus human effectiveness in essay evaluation: ChatGPT as an automated essay scoring tool. Discover Education, 1(1), 1-12. https:// doi. org/ 10. 1007/ s44234-024-00002-z · DOI
  6. Burner, T., Lindvig, Y., & Wærness, J. I. (2025). "We should not be like a dinosaur"-Using AI technologies to provide forma- tive feedback to students. Education Sciences, 15(1), 58.
  7. Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y. S., Gašević, D., & Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers And Education: Artificial Intelligence,2, Article 100027.
  8. Chen, W., Lyu, Q., & Su, J. (2025). How more-improvement and less-improvement groups differ in peer feedback giving and receiving practice-an exploratory study. Instructional Science, 53(1), 27-47.
  9. Diezmann, C. M., & Watters, J. J. (2015). The knowledge base of subject matter experts in teaching: A case study of a pro- fessional scientist as a beginning teacher. International Journal of Science and Mathematics Education,13, 1517-1537. https:// doi. org/ 10. 1007/ s10763-014-9561-x · DOI
  10. Er, E., Akçapınar, G., Bayazıt, A., Noroozi, O., & Banihashem, S. K. (2024). Assessing student perceptions and use of instructor versus AI-generated feedback. British Journal of Educational Technology,56(3), 1079-91.
  11. Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2024). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in education and teaching international,61(3), 460-474.
  12. Ferretti, R. P., MacArthur, C. A., & Dowdy, N. S. (2000). The effects of an elaborated goal on the persuasive writing of stu- dents with learning disabilities and their normally achieving peers. Journal of Educational Psychology, 92(4), 694.
  13. Fokides, E., & Peristeraki, E. (2024). Comparing ChatGPT's correction and feedback comments with that of educators in the context of primary students'short essays written in English and Greek. Education and Information Technologies, 30(2), 2577-5621. https:// doi. org/ 10. 1007/ s10639-024-12912-8 · DOI
  14. Gan, W., Qi, Z., Wu, J., & Lin, J. C. W. (2023). Large language models in education: Vision and opportunities. In Proceedings of the IEEE International Conference on Big Data. pp. 4776-4785 https:// doi. org/ 10. 1109/ BigDa ta2023. 4776-4785 · DOI
  15. Gombert, S., Fink, A., Giorgashvili, T., Jivet, I., Di Mitri, D., Yau, J., ... & Drachsler, H. (2024). From the automated assessment of student essay content to highly informative feedback: A case study. International Journal of Artificial Intelligence in Education, 34(4), 1378-1416.
  16. Guo, S., Latif, E., Zhou, Y., Huang, X., & Zhai, X. (2024). Using generative AI and multi-agents to provide automatic feedback [Preprint]. arXiv. https:// doi. org/ 10. 48550/ arXiv. 2411. 07407 · DOI
  17. Irvin, M. K., Asaba, M., Stegall, J., Frank, M., & Gweon, H. (2021). "This one's great! That one's okay. ": Investigating the role of selective vs. indiscriminate praise on children's learning behaviors. The undergraduate research journal of psychology at UCLA, 8, 50-82.
  18. Jamal, R., & Renzella, J. (2024). Enhancing formative feedback at scale with the intelligent feedback assistant. In SIGCSE 2024 -Proceedings of the 55th ACM Technical Symposium on Computer Science Education, 2:1692-1693. https:// doi. org/ 10. 1145/ 36262 53. 36354 82 · DOI
  19. Jansen, T., Hoft, L., Bahr, L., Fleckenstein, J., Moller, J., Koller, O., & Meyer, J. (2024). Comparing generative AI and expert feedback to students' writing: Insights from student teachers. Psychologie in Erziehung und Unterricht, 71(2), 80-92. https:// doi. org/ 10. 2378/ peu20 24. art08d · DOI
  20. Kaldaras, L., Hoffman, M., & Glassman, M. (2024). Developing valid assessments in the era of generative artificial intel- ligence. Frontiers in Education, 9, 1399377. https:// doi. org/ 10. 3389/ feduc. 2024. 13993 77 · DOI
  21. Krause, M., Garncarz, T., Song, J., Gerber, E. M., Bailey, B. P., & Dow, S. P. (2017). Critique style guide: Improving crowd- sourced design feedback with a natural language model. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 4627-4639. Association for Computing Machinery. https:// doi. org/ 10. 1145/ 30254 53. 30258 · DOI
  22. Lee, H.-S., Pallant, A., Pryputniewicz, S., Lord, T., Mulholland, M., & Liu, O. L. (2019). Automated text scoring and real-time adjustable feedback: Supporting revision of scientific arguments involving uncertainty. Science Education,103(3), 590-622. https:// doi. org/ 10. 1002/ sce. 21504 · DOI
  23. Mohammed, S. J., & Khalid, M. W. (2025). Under the world of AI-generated feedback on writing: Mirroring motivation, foreign language peace of mind, trait emotional intelligence, and writing development. Language Testing in Asia, 15, 7. https:// doi. org/ 10. 1186/ s40468-025-00343-2 · DOI
  24. Moon, J. A. (2006). Learning journals: A handbook for reflective practice and professional development. Routledge.
  25. Nazari, M., & Saadi, G. (2024). Developing effective prompts to improve communication with ChatGPT: A formula for higher education stakeholders. Discover Education, 3(1), 45.
  26. Nazaretsky, T., Mejia-Domenzain, P., Swamy, V., Frej, J., Käser, T. (2024). AI or human? Evaluating student feedback percep- tions in higher education. In: Ferreira Mello, R., Rummel, N., Jivet, I., Pishtari, G., Ruipérez Valiente, J.A. (eds) Technol- ogy Enhanced Learning for Inclusive and Equitable Quality Education. EC-TEL 2024. Lecture Notes in Computer Science, vol 15159. Springer, Cham. https:// doi. org/ 10. 1007/ 978-3-031-72315-5_ 20 · DOI · Paper page
  27. Nicol, D. (2014). From monologue to dialogue: Improving written feedback processes in mass higher education. In Approaches to assessment that enhance learning in higher education (pp. 11-27). Routledge.
  28. Ryan, M. (2011). Improving reflective writing in higher education: A social semiotic perspective. Teaching in higher educa- tion, 16(1), 99-111. https:// doi. org/ 10. 1080/ 13562 517. 2010. 507311 · DOI
  29. Ruwe, T., & Mayweg-Paus, E. (2024). Embracing LLM feedback: The role of feedback providers and provider information for feedback effectiveness. Frontiers in Education, 9, 1461362. https:// doi. org/ 10. 3389/ feduc. 2024. 14613 62 · DOI
  30. Scarlatos, A., Smith, D., Woodhead, S., & Lan, A. (2024, July). Improving the validity of automatically generated feedback via reinforcement learning. In: International Conference on Artificial Intelligence in Education (pp. 280-294). Cham: Springer Nature Switzerland. https:// doi. org/ 10. 1007/ 978-3-031-64302-6_ 20 · DOI
  31. Shi, M. (2019). The effects of class size and instructional technology on student learning performance. The International Journal of Management Education, 17(1), 130-138. https:// doi. org/ 10. 1016/j. ijme. 2019. 01. 004. · DOI · Paper page
  32. Tirumala, K., Markosyan, A., Zettlemoyer, L., & Aghajanyan, A. (2022). Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35, 38274-38290.
  33. Venter, J., Coetzee, S. A., & Schmulian, A. (2024). Exploring the use of artificial intelligence (AI) in the delivery of effective feedback. Assessment & Evaluation in Higher Education,50(4), 516-536. https:// doi. org/ 10. 1080/ 02602 938. 2024. 24156 · DOI
  34. Vinutha, D. C., Kavyashree, S., Vijay, C. P., & Raju, G. T. (2022). Innovative practices in education systems using artificial intel- ligence for advanced society. In The New Advanced Society. https:// doi. org/ 10. 1002/ 97811 19884 392. · DOI
  35. ch16 · DOI
  36. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824-24837.
  37. Williyan, A., Ilyas, M., Shahat, S., Guntur, M., & Rosalina, U. (2025). AI-powered pedagogy for argumentative essays in tertiary ELT: Insights from focus groups. Journal of General Education and Humanities,4(3), 671-686. https:// doi. org/ 10. 58421/ gehu. v4i3. 423 · DOI
  38. Ye, Q., Axmed, M., Pryzant, R., & Khani, F. (2023). Prompt engineering a prompt engineer [Preprint]. arXiv. https:// doi. org/ 10. 48550/ arXiv. 2311. 05661 · DOI
  39. Yesilyurt, Y. E. (2023). AI-enabled assessment and feedback mechanisms for language learning: Transforming pedagogy and learner experience. In Transforming the Language Teaching Experience in the Age of AI. https:// doi. org/ 10. 4018/ 978-1-6684-9893-4. · DOI
  40. ch002 · DOI
  41. Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023). Why Johnny can't prompt: How non-AI experts try (and fail) to design LLM prompts. In: Proceedings of the 2023 CHI conference on human factors in computing systems (pp. 1-21). https:// doi. org/ 10. 1145/ 35445 48. 35813 · DOI
  42. Zhang, J., & Wang, J. (2025). Student perceptions of hybrid feedback: Using gen-AI to enhance engagement with EAP writing feedback. The JALT CALL Journal,21(2), 1-27. https:// doi. org/ 10. 29140/ jaltc all. v21n2. 2175 · DOI
  43. Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022. Large language models are humanlevel prompt engineers. In: The Eleventh International Conference on Learning Representations.

Cited by

Sorted by citations

Authors
Jovic, M., Papakonstantinidis, S., & Kirkpatrick, R.
Affiliations
  • Gulf University for Science & Technology (KW)
  • Central College (US)
Corresponding author
Jovic, Marina
ORCID
Venue
Language Testing in Asia
Publisher
Springer Science and Business Media LLC
Published
2025-10-14
Volume
15
Issue
1
ISSN
2229-0443 (ISSN-L)
DOI
10.1186/s40468-025-00389-2
OpenAlex
W4415157607
Cited by
11
References
9
Citations by year
2026: 11
FWCI
11.53
Type
journal article
Language
en
Open access
Licence
Locations
  • Language Testing in Asia (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Social Sciences › Education › Student Assessment and Feedback
  • Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
Keywords
large language models, AI-generated feedback, student writing assessment, argumentative essay, reflective essays, assessment rubrics, multilingual learners, Claude 3.7 Sonnet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review