On Automated Essay Grading using Large Language Models

Liew, Pei Yee; Tan, Ian K. T.

doi:10.1145/3709026.3709030

Cited by 42Educational Technology and AssessmentNatural Language Processing TechniquesArtificial Intelligence in Education

On Automated Essay Grading using Large Language Models

Open access

Liew, Pei Yee, Tan, Ian K. T.·2024·Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence

DOI 10.1145/3709026.3709030

Open this paper in CoChat — free account

Integrity signals

Registry

Crossref · 10.1145/3709026.3709030

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Modifying grading instructions through prompts did not substantially alter GPT-4's performance on essay evaluation.
“some research has indicated that modifying grading instructions through prompts did not substantially alter GPT-4's performance on essay evaluation (Liew and Tan, 2024).”
From A Multi-Dimensional Decision Framework for Evaluating and Selecting AI-Enabled Instructional Technology in U.S. Higher Education (2025) · cites this paper as Liew and Tan, 2024

The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
The Power of Feedback
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · see citing works
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1× · see citing works
“This mechanism evaluates the relevance of input elements during prediction [16].”
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · see citing works
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279 · see citing works
Exploring the potential of using an AI language model for automated essay scoring
Mizumoto, A., & Eguchi, M. (2023) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2023.100050 · Cited by 647 · see citing works
“A neural approach to automated essay scoring,”
Taghipour, K., & Ng, H. T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1193 · Cited by 537 · see citing works
Comparing the quality of human and ChatGPT feedback of students’ writing
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. B. (2024) · Learning and Instruction · doi:10.1016/j.learninstruc.2024.101894 · Cited by 509 · cited 1× · see citing works
“A study by [33] explored the potential of Generative AI, specifically ChatGPT, to provide formative feedback comparable to human raters for student compositions.”
Automatic Text Scoring Using Neural Networks
Alikaniotis, D., Yannakoudakis, H., & Rei, M. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1068 · Cited by 323 · see citing works
An Overview of Current Research on Automated Essay Grading
Valenti, S., Neri, F., & Cucchiarelli, A. (2003) · Journal of Information Technology Education Research · doi:10.28945/331 · Cited by 313 · see citing works
Automatic Text Scoring Using Neural Networks
Alikaniotis, D., Yannakoudakis, H., & Rei, M. (2016) · Apollo (University of Cambridge) · doi:10.17863/cam.376 · Cited by 267 · see citing works
Dependence of Weighted Kappa Coefficients on the Number of Categories
Brenner, H., & Kliebsch, U. (1996) · Epidemiology · doi:10.1097/00001648-199603000-00016 · Cited by 255 · cited 1× · see citing works
“QWK’s ability to account for the degree of disagreement makes it suitable for an ordinal variable [3].”
Computer Grading of Student Prose, Using Modern Concepts and Software
Page, E. B. (1994) · The Journal of Experimental Education · doi:10.1080/00220973.1994.9943835 · Cited by 228 · see citing works
Automatic Features for Essay Scoring – An Empirical Study
Dong, F., & Zhang, Y. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1115 · Cited by 212 · see citing works
Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression
Phandi, P., Chai, K. M. A., & Ng, H. T. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1049 · Cited by 171 · see citing works
Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking
Yang, R., Cao, J., Wen, Z., Wu, Y., & He, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.141 · Cited by 146 · see citing works
Proceedings of the 14th International Conference on Educational Data Mining
Hsiao, I., Sahebi, S., Bouchet, F., & Vie, J.-J. (2021) · HAL · Cited by 98 · see citing works
Automated Essay Feedback Generation and Its Impact on Revision
Liu, M., Yi, L., Xu, W., & Liu, L. (2016) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2016.2612659 · Cited by 79 · see citing works
Robust Neural Automated Essay Scoring Using Item Response Theory
Uto, M., & Okano, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-52237-7_44 · Cited by 47 · see citing works
Assistance and Feedback Mechanism in an Intelligent Tutoring System for Teaching Conversion of Natural Language into Logic
Perikos, I., Grivokostopoulou, F., & Hatzilygeroudis, I. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0139-y · Cited by 44 · see citing works
Adaptive feedback generation to support teachers in web-based distance education
Kosba, E., Dimitrova, V. G., & Boyle, R. (2007) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9031-z · Cited by 42 · see citing works
Automated Language Scoring System by Employing Neural Network Approaches
Kwong, A., Muzamal, J. H., & Khan, M. U. G. (2019) · International Conference on Emerging Technologies · doi:10.1109/icet48972.2019.8994673 · Cited by 4 · see citing works
Feature Group Importance for Automated Essay Scoring
Tan, J. S., & Tan, I. K. T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80253-0_6 · Cited by 3 · see citing works
Self-citation

Flagged first, then by citations

Sorted by citations

A comprehensive review of AI-powered grading and tailored feedback in universities
Deepshikha, D. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00517-0 · Cited by 34 · see citing works
Self-assessment accuracy in the age of artificial Intelligence: Differential effects of LLM-generated feedback
Liebenow, L. W., Schmidt, F. T. C., Meyer, J., & Fleckenstein, J. (2025) · Comput. Educ. · doi:10.1016/j.compedu.2025.105385 · Cited by 22 · see citing works
Enhancing a large language model with a chain-of-metacognitive reasoning approach increases argumentative writing evaluation accuracy, student writing outcomes, and mental effort
Wang, S., & Hew, K. F. (2026) · Computers & Education · doi:10.1016/j.compedu.2026.105621 · Cited by 13 · see citing works
Automatic Scoring of Elementary School Essays in Brazilian Portuguese with LLMs: Comparing Gemini, GPT-4o, Claude, and Mistral
Lobo, J., Anthony, L., Falcão, A., Xavier, C., da Costa, N. T., Isotani, S., Ibert, I., Rodrigues, L., & Mello, R. F. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12161 · Cited by 7 · see citing works
Evaluating large language models for AI-assisted grading: a framework and case study in higher education
Sáez, Y., Garcia, L. M., Mochón, A., & Isasi, P. (2026) · Scientific Reports · doi:10.1038/s41598-026-48656-3 · Cited by 6 · see citing works
Breaking the Ceiling: Mitigating Extreme Response Bias in Surveys Using an Open-Ended Adaptive-Testing System and LLM-Based Response Analysis
Gish, M., Nowominski, A., & Dror, R. (2026) · AI · doi:10.3390/ai7020073 · Cited by 3 · see citing works
Can ChatGPT score ESL writing? A correlation analysis between teacher and GenAI scores
Alkhofi, A. (2026) · Language Teaching Research · doi:10.1177/13621688261415584 · Cited by 4 · see citing works
A Unified Conceptual Hybrid Framework for the Automated Assessment of Short Answers
Ramnarain-Seetohul, V., Rosunally, Y. Z., & Bassoo, V. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00487-5 · Cited by 3 · see citing works
Beyond the Black Box: Interpretable Multi-Trait Essay Scoring with Trait-Aware Transformer
Tang, X. (2026) · Electronics · doi:10.3390/electronics15051066 · Cited by 3 · see citing works
Source integration in source-based writing: From theoretical modeling to predictive analytics
Potter, A., Tian, Y., Balyan, R., Goldshtein, M., Allen, L. K., & McNamara, D. S. (2026) · Assessing Writing · doi:10.1016/j.asw.2026.101081 · Cited by 3 · see citing works
Optimizing LLMs for Automated Essay Scoring: Finetuning and Systematic Prompting
ElMassry, A. M., & Zaki, N. (2025) · IEEE International Conference on Application of Information and Communication Technologies (AICT) · doi:10.1109/aict67988.2025.11268740 · Cited by 1 · see citing works
A Multi-Dimensional Decision Framework for Evaluating and Selecting AI-Enabled Instructional Technology in U.S. Higher Education
Akter, F., & Parvatha Reddy, K. K. (2025) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2025.v07i06.84921
“…the efficacy of prompt engineering is highly dependent on specific tasks and the complexity of evaluation criteria, and some research has indicated that modifying grading instructions through prompts did not substantially alter GPT-4’s performance on essay evaluation (Liew and Tan, 2024).”
Influential
AI-Supported Feedback for Writing
Kemal, S., Süzan, D., Öztüfekçi, A., & Eşfer, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7579-3.ch011
Rubric-based Automated Essay Scoring and Providing Insightful Feedback by Utilizing LLM for Indonesian Language Academic Text
Sibarani, R. G., Hidayah, I., & Permanasari, A. E. (2025) · International Conference on Information Technology and Computing (ICITCOM) · doi:10.1109/icitcom66635.2025.11264872 · Cited by 1 · see citing works
Türkçe ve İngilizce Anahtar Kelime Üretimi İçin Özgün Veri Seti Oluşturulması ve Büyük Dil Modellerine İnce Ayar Uygulanması
Arslan, Ş. N., Görmez, Y., & Arslan, H. (2026) · Bilgisayar Bilimleri ve Mühendisliği Dergisi · doi:10.54525/bbmd.1838196
Agentic AI Framework for Low-Resource Essay Evaluation via Scoring, Explanation, and Debate
Thapa, S., Rauniyar, K., Shiwakoti, S., Adhikari, S., Rashid, J., Kim, J., & Naseem, U. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402339
TBERT: Bridging Text Generation and Score Regression Through Hierarchical Feature Fusion Based LLM for Automated Essay Scoring
Li, X., Zhang, H., Yang, S., Wang, Y. Y., Liu, X., & Lu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7072-0_35
CORTEX: A Capability-Driven Reasoning Framework for Zero-Shot Large Language Model Selection
Zhao, K., Liu, Z., Wang, Z., Wang, X., Yang, H., Wang, N., Wang, K., & Lian, S. (2025) · International Conference on Computer Science and Artificial Intelligence · doi:10.1145/3788149.3788171
Future Concepts of Artificial Intelligence
ŞAHİN, H. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/406016
Chain of Thought Guided Fine-Tuning of Large Language Models for Rubric-Assisted Automated Essay Grading
Xuan, T. N., Thanh, T. N., Viet, V. H., & Quynh, T. D. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18316-3_27
Grading Machines: Can AI Exam-Grading Replace Law Professors?
Cope, K. L., Frankenreiter, J., Hirst, S., Posner, E. A., Schwarcz, D. B., & Thorley, D. (2026) · Journal of law & empirical analysis. · doi:10.1177/2755323x261434265
Open-ended Structured Question Assessment with Human-LLM Collaboration
Lin, F., Lin, Y., Cao, K., Deng, Z., & Cai, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791034
MADCAP: A Multi-agent Deliberative Framework for Robust Assessment of Open-Ended Questions
Lin, S., He, W., & Li, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_32
Automated Essay Scoring Via Llm-Generated Comments, Bert Intermediate Layers, and Cognitive Linguistic Features
Peng, Y., & Zhang, Y. (2025) · International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir67391.2025.11491043
AI Agent-Based Automated Essay Marking Framework Using Large Language Models
Thenura, S., Asanka, D., Mahanama, T. V., & Rajapakse, C. (2026) · International Research Conference on Smart Computing and Systems Engineering (SCSE) · doi:10.1109/scse70081.2026.11499850
Comparative Analysis of Zero-Shot Testing on Different LLMs for Automated Grading Systems for Business Education
Abdul-Fattah, K., Khoriba, G. A., & Al‐Atabany, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-19865-5_10
How Prompting and Context Granularity Shape LLM-Based Assessment of Academic Writing: A Case Study in Serbian
Francišković, T. S., Сливка, Ј., Luburić, N., & Perovic, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29755-6_52
Do Language Models Matter? Evaluating Model Choice in a Multi-agent Feedback Framework
Kolle, H. N., Epp, C. D., Liaqat, A., & Cutumisu, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-29788-4_16
An AI Tutor System with a Virtual Whiteboard for Statistics Learning
Tran, T., Boussé, M., & Härmä, A. (2026) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon67543.2026.11574422
Leveraging Large Language Models and Speech Transformers for Automated Just-A-Minute Speech Scoring
Karthiyayini, S., Priya, B. B. V., Ravichandra, T., Santhoshkumar, S. P., Vinodhini., K., & Kaliappan, A. T. (2026) · International Conference on System, Computation, Automation and Networking (ICSCAN) · doi:10.1109/icscan66520.2026.11588313
Automated Test Marking Using Multi-modal AI Techniques: A Systematic Literature Review
Musafiri, F., Olaifa, M., & Du, C. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-25187-9_9
AI-Powered Assessment Systems: Innovations and Ethical Considerations in Education
Thagaram, E., & Sasikala, R. (2026) · Springer eBooks · doi:10.1007/978-981-92-1746-5_1
Is One Shot Prompting Enough for Reliable Essay Evaluation by Large Language Models?
Ilijoski, B., & Zdravkova, K. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-24513-7_36
Intelligent Assessment and Tutoring System Design Based on Large Language Models
Zhang, L., & Jia, J. (2026) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste69562.2026.11649662
Risks, Mitigations and Challenges of LLM-AS-A-Judge in Automated Scoring
Li, Y., Sun, Y., & Hou, S. (2026) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste69562.2026.11647786
Benchmarking SLM and LLM in Feature Generation for Interpretable Automated Essay Scoring
Chovanec, K., Fields, J., & Madiraju, P. (2026) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri69576.2026.00023
Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis
Alghamdi, L. H., & Alghizzi, T. M. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1908592
InsightGrader: Intelligent Essay/Answer Evaluation with Misconception Tracking and Cognitive Load Insights - A Comparative Analysis
Sai, K. T., Gnaneswar, R., Kushal, U. K., & Devi, B. S. (2026) · International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC) · doi:10.1109/assic70037.2026.11717269
Towards Reliable Automated Essay Scoring: A Feature‐Calibrated LLM Approach for the IELTS Exam
Aghadadi, M. S., & Faili, H. (2026) · Applied Computational Intelligence and Soft Computing · doi:10.1155/acis/5145886
An Automated Exam Marking Framework using an Artificial Intelligence Tool for English Writing in General Foundation Program
Ahmad, N. (2026) · International Journal of Teacher Education Research Studies (IJTERS) · doi:10.63090/ijters/3049.1614.0047
A GenAI-supported job interview training robot for young adults with special education needs
Fung, K. Y. (2026) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2026.2748786
SLMs como Avaliadores Automatizados: Avaliando Competências de Escrita Narrativa com Modelos Adaptados por LoRA
Batista, H., Anthony, L., FALCÃO, A., Alves, G., Batista, M. D. C. M., do Nascimento, A. C. A., & Mello, R. F. (2026) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2026.27403

Authors: Liew, P. Y., & Tan, I. K. T.
Affiliations: Heriot-Watt University Malaysia (MY)
ORCID: Liew, Pei Yee
Tan, Ian K. T.
Venue: Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence
Publisher: ACM
Published: 2024-12-06
Pages: 204-211
DOI: 10.1145/3709026.3709030
OpenAlex: W4407601416
Cited by: 41
References: 22
Citations by year: 2026: 31 · 2025: 10
FWCI: 15.61
Type: conference paper
Language: en
Open access: gold
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0/ (vor)
Topics: Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords: automated essay scoring, large language models, essay scoring, natural language processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

On Automated Essay Grading using Large Language Models

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks