Applying large language models and chain-of-thought for automatic scoring
Open accessAbstract
This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics, developed based on a novel approach, WRVRT (prompt writing, reviewing, validating, revising, and testing). Results indicated that few-shot (acc = 0.67) outperformed zero-shot learning (acc = 0.60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = 0.60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT -3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches. This study also demonstrates the potential of LLMs in facilitating explainable and interpretable automatic scoring, emphasizing that CoT enhances accuracy and transparency, particularly when used with item stem and scoring rubrics.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Hallucinations remain a persistent limitation of LLMs despite technical interventions such as chain-of-thought and retrieval-augmented generation prompting.
“Despite considerable technical interventions aimed at reducing hallucinations-such as chain-of-thought and retrieval-augmented generation prompting (Gao et al. 2023;Lee et al. 2024)-hallucinations remain a persistent limitation of LLMs.”
From Bridging risk and retention: risk coping with privacy and hallucination in AI (2026) · cites this paper as Lee et al. 2024
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Baidoo-Anu, D., & Owusu Ansah, L. (2023) · doi:10.2139/ssrn.4337484 · Cited by 3,004 · see citing works
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951 · see citing works
BAİDOO-ANU, D., & OWUSU ANSAH, L. (2023) · Journal of AI · doi:10.61969/jai.1337500 · Cited by 1,826 · see citing works
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Techniques for interpretable machine learning
Du, M., Liu, N., & Hu, X. (2019) · Communications of the ACM · doi:10.1145/3359786 · Cited by 1,268 · see citing works
Self-citationTree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158 · see citing works
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080 · see citing works
Explainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556 · see citing works
An automated essay scoring systems: a systematic literature review
Ramesh, D., & Sanampudi, S. K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10068-2 · Cited by 550 · see citing works
Explainable AI Methods - A Brief Overview
Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_2 · Cited by 480 · see citing works
C-rater: Automated Scoring of Short-Answer Questions
Leacock, C., & Chodorow, M. S. (2003) · Computers and the Humanities · doi:10.1023/a:1025779619903 · Cited by 384 · see citing works
Learning to work with the black box: Pedagogy for a world with artificial intelligence
Bearman, M., & Ajjawi, R. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13337 · Cited by 346 · see citing works
Applying machine learning in science assessment: A systematic review
Zhai, X., Yin, Y., Pellegrino, J. W., Haudek, K. C., & Shi, L. (2020) · Studies in Science Education · doi:10.1080/03057267.2020.1735757 · Cited by 257 · see citing works
A Review on Word Embedding Techniques for Text Classification
Birunda, S. S., & Devi, R. K. (2021) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-15-9651-3_23 · Cited by 224 · see citing works
Lee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., & Kim, H. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12249-8 · Cited by 217 · see citing works
MathPrompter: Mathematical Reasoning using Large Language Models
Imani, S., Du, L., & Shrivastava, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.4 · Cited by 212 · see citing works
Organisciak, P., Acar, S., Dumas, D., & Berthiaume, K. (2023) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2023.101356 · Cited by 202 · see citing works
Fine-tuning ChatGPT for automatic scoring
Latif, E., & Zhaı, X. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100210 · Cited by 186 · see citing works
Self-citationContrastive Decoding: Open-ended Text Generation as Optimization
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J. M., Hashimoto, T., Zettlemoyer, L., & Lewis, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.687 · Cited by 179 · see citing works
Chatgpt for next generation science learning
Zhai, X. (2023) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/3589649 · Cited by 178 · see citing works
Nehm, R. H., Ha, M., & Mayfield, E. (2011) · Journal of Science Education and Technology · doi:10.1007/s10956-011-9300-9 · Cited by 150 · see citing works
From substitution to redefinition: A framework of machine learning‐based science assessment
Zhaı, X., Haudek, K. C., Shi, L., Nehm, R. H., & Urban‐Lurain, M. (2020) · Journal of Research in Science Teaching · doi:10.1002/tea.21658 · Cited by 134 · see citing works
Automated essay scoring with string kernels and word embeddings
Cozma, M., Butnaru, A. M., & Ionescu, R. T. (2018) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p18-2080 · Cited by 127 · see citing works
Applying machine learning to automatically assess scientific models
Zhaı, X., He, P., & Krajcik, J. S. (2022) · Journal of Research in Science Teaching · doi:10.1002/tea.21773 · Cited by 127 · see citing works
A Contrastive Framework for Neural Text Generation
Su, Y., Lan, T., Wang, Y., Yogatama, D., Kong, L., & Collier, N. (2022) · neural information processing systems · doi:10.52202/068431-1566 · Cited by 126 · see citing works
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. J. L., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.82 · Cited by 117 · see citing works
Hahn, M. G., Navarro, S. M. B., De La Fuente Valentin, L., & Burgos, D. (2021) · IEEE Access · doi:10.1109/access.2021.3100890 · Cited by 111 · see citing works
Language models and Automated Essay ScoringPreprint
Rodriguez, P. U., Jafari, A., & Ormerod, C. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09482 · Cited by 106 · see citing works
Unpacking the "Black Box" of AI in EducationPreprint
Gillani, N., Eynon, R., Chiabaut, C., & Finkel, K. (2022) · arXiv (Cornell University) · doi:10.30191/ets.202301_26(1).0008 · Cited by 105 · see citing works
Meister, C., Pimentel, T., Wiher, G., & Cotterell, R. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00536 · Cited by 95 · see citing works
A Contrastive Framework for Neural Text GenerationPreprint
Su, Y., Lan, T., Wang, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06417 · Cited by 83 · see citing works
Bewersdorff, A., Seßler, K., Baur, A., Kasneci, E., & Nerdel, C. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100177 · Cited by 76 · see citing works
A review of research on machine learning in educational technology
Korkmaz, C., & Correia, A. (2019) · Educational Media International · doi:10.1080/09523987.2019.1669875 · Cited by 73 · see citing works
Jescovitch, L. N., Scott, E. E., Cerchiara, J. A., MERRILL, J. E., Urban‐Lurain, M., Doherty, J. H., & Haudek, K. C. (2020) · Journal of Science Education and Technology · doi:10.1007/s10956-020-09858-0 · Cited by 73 · see citing works
Context Matters: A Strategy to Pre-train Language Model for Science Education
Liu, Z., He, X., Liu, L., Liu, T., & Zhaı, X. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-36336-8_103 · Cited by 54 · see citing works
A Theoretical Analysis of the Repetition Problem in Text Generation
Fu, Z., Lam, W. P., So, A. M., & Shi, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17520 · Cited by 53 · see citing works
Binding Language Models in Symbolic Languages
Cheng, Z., Xie, T., Shi, P., Li, C., Nadkarni, R., Hu, Y., Xiong, C., Radev, D., Ostendorf, M., Zettlemoyer, L., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02875 · Cited by 53 · see citing works
Assessing Argumentation Using Machine Learning and Cognitive Diagnostic Modeling
Zhaı, X., Haudek, K. C., & Ma, W. (2022) · Research in Science Education · doi:10.1007/s11165-022-10062-w · Cited by 51 · see citing works
Using automated analysis to assess middle school students' competence with scientific argumentation
Wilson, C. D., Haudek, K. C., Osborne, J. F., Bracey, Z. B., Cheuk, T., Donovan, B., Stuhlsatz, M. A. M., Santiago, M. M., & Zhaı, X. (2023) · Journal of Research in Science Teaching · doi:10.1002/tea.21864 · Cited by 49 · see citing works
Uses of Artificial Intelligence in STEM Education
(2024) · doi:10.1093/oso/9780198882077.001.0001 · Cited by 49 · see citing works
MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
Shen, J. T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., Graff, B., & Lee, D. (2021) · arXiv · doi:10.48550/arxiv.2106.07340 · Cited by 47 · see citing works
Self-citationZhaı, X., Haudek, K. C., Stuhlsatz, M. A. M., & Wilson, C. D. (2020) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2020.100916 · Cited by 40 · see citing works
Gemini Pro Defeated by GPT-4V: Evidence from EducationPreprint
Lee, G.-G., Latif, E., Shi, L.-H., & Zhai, X. (2023) · arXiv.org · doi:10.48550/arxiv.2401.08660 · Cited by 40 · see citing works
Self-citationChatGPT and AI: The Game Changer for Education
Zhai, X. (2023) · doi:10.13140/rg.2.2.31107.37923 · Cited by 40 · see citing works
Self-citationWu, X., He, X., Liu, T., Liu, N., & Zhaı, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_33 · Cited by 34 · see citing works
Self-citationSurvey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to TransformersPreprint
Haller, S., Aldea, A., Seifert, C., & Strisciuglio, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.03503 · Cited by 31 · see citing works
Truncation Sampling as Language Model Desmoothing
Hewitt, J., Manning, C. D., & Liang, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.249 · Cited by 31 · see citing works
Advancing automatic guidance in virtual science inquiry: from ease of use to personalization
Zhaı, X. (2021) · Educational Technology Research and Development · doi:10.1007/s11423-020-09917-8 · Cited by 25 · see citing works
Muhamedyev, R. I., Yakunin, K., Kuchin, Y. I., Сымагулов, А., Buldybayev, T., Murzakhmetov, S., & Abdurazakov, A. (2020) · Cogent Engineering · doi:10.1080/23311916.2020.1769349 · Cited by 22 · see citing works
Gemini Pro Defeated by GPT-4V: Evidence from EducationPreprint
Lee, G.-G., Latif, E., Shi, L., & Zhaı, X. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.17393.25448/1 · Cited by 17 · see citing works
Self-citationChen, Z., Zhou, Q., Shen, Y., Hong, Y., Zhang, H., & Gan, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.05226 · Cited by 8 · see citing works
G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in GermanPreprint
Latif, E., Lee, G.-G., Neumann, K., Kastorff, T., & Zhaı, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.06584 · Cited by 3 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Using ChatGPT for Science Learning: A Study on Pre-service Teachers' Lesson Planning
Lee, G.-G., & Zhaı, X. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3401457 · Cited by 225 · see citing works
Self-citationZhai, X. (2025) · Journal of Science Education and Technology · doi:10.1007/s10956-024-10174-0 · Cited by 184 · see citing works
Prompt Engineering in Education: A Systematic Review of Approaches and Educational Applications
Qian, Y. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251365189 · Cited by 91 · see citing works
Flodén, J. (2024) · British Educational Research Journal · doi:10.1002/berj.4069 · Cited by 110 · see citing works
Seßler, K., Fürstenberg, M., Bühler, B., & Kasneci, E. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706527 · Cited by 84 · see citing works
Saleh, M. I. (2025) · Journal of Hospitality Marketing & Management · doi:10.1080/19368623.2025.2458603 · Cited by 80 · see citing works
On the use of large language models in model-driven engineering
Rocco, J. D., Ruscio, D. D., Sipio, C. D., Nguyen, P. T., & Rubei, R. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01263-8 · Cited by 73 · see citing works
Tang, X., Chen, H., Lin, D., & Li, K. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e34262 · Cited by 79 · see citing works
Beyond Traditional Assessment: Exploring the Impact of Large Language Models on Grading Practices
Fagbohun, O., Iduwe, N. P., Abdullahi, M., Ifaturoti, A., & Nwanna, O. (2024) · Journal of Artificial Intelligence, Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/19 · Cited by 68 · see citing works
RETRACTED: The impact of generative AI on essay revisions and student engagement
Lo, N., Wong, A., & Chan, S. (2025) · Computers and Education Open · doi:10.1016/j.caeo.2025.100249 · Cited by 78 · see citing works
RetractedChan, S. T. S., Lo, N. P. K., & Wong, A. M. H. (2024) · Contemporary Educational Technology · doi:10.30935/cedtech/15607 · Cited by 56 · see citing works
Generative AI in education: ChatGPT-4 in evaluating students’ written responses
Jauhiainen, J. S., & Guerra, A. G. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2024.2422337 · Cited by 63 · see citing works
Shi, L. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13684-5 · Cited by 46 · see citing works
Generative AI risks and resilience: How users adapt to hallucination and privacy challenges
Lee, C., Kim, J., Lim, J. S., & Shin, D. (2025) · Telematics and Informatics Reports · doi:10.1016/j.teler.2025.100221 · Cited by 53 · see citing works
Large Language Models as Molecular Design Engines
Bhattacharya, D., Cassady, H. J., Hickner, M. A., & Reinhart, W. F. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01396 · Cited by 44 · see citing works
Rasool, Z., Kurniawan, S., Balugo, S., Barnett, S., Vasa, R., Chesser, C., Hampstead, B. M., Belleville, S., Mouzakis, K., & Bahar-Fuchs, A. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100083 · Cited by 34 · see citing works
ChatGPT as an Instructor’s Assistant for Generating and Scoring Exams
Fernández, A., López-Torres, M., Fernández, J. J., & Vázquez-García, D. (2024) · Journal of Chemical Education · doi:10.1021/acs.jchemed.4c00231 · Cited by 42 · see citing works
Examining the responsible use of zero-shot AI approaches to scoring essays
Johnson, M., & Zhang, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-79208-2 · Cited by 33 · see citing works
Chan, S., Lo, N., & Wong, A. (2024) · Cogent Education · doi:10.1080/2331186x.2024.2440182 · Cited by 43 · see citing works
RetractedAutomatic item generation in various STEM subjects using large language model prompting
Chan, K. W., Ali, F., Park, J., Sham, K. S. B., Tan, E. Y. T., Chong, F. W. C., Qian, K., & Sze, G. K. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100344 · Cited by 38 · see citing works
Farrokhnia, M., Latifi, S., Papadopoulos, P. M., Hogenkamp, L., Gijlers, H., Khosravi, H., & Noroozi, O. (2026) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-026-00579-9 · Cited by 41 · see citing works
Chen, Z., & Wan, T. (2025) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.21.010126 · Cited by 33 · see citing works
Can AI support human grading? Examining machine attention and confidence in short answer scoring
Li, Y., Raković, M., Srivastava, N., Li, X., Guan, Q., Gašević, D., & Chen, G. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105244 · Cited by 29 · see citing works
Polat, M., Karataş, İ. H., & Varol, N. (2025) · Leadership and Policy in Schools · doi:10.1080/15700763.2024.2412204 · Cited by 28 · see citing works
Zhaı, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4970687 · Cited by 28 · see citing works
Chan, S., Lo, N., & Wong, A. (2024) · rEFLections · doi:10.61508/refl.v31i3.277514 · Cited by 29 · see citing works
Applying Generative Artificial Intelligence to Critiquing Science Assessments
Nguyen, H., & Hayward, J. (2024) · Journal of Science Education and Technology · doi:10.1007/s10956-024-10177-x · Cited by 32 · see citing works
Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AI
Lee, G.-G., & Zhaı, X. (2025) · TechTrends · doi:10.1007/s11528-024-01035-z · Cited by 27 · see citing works
Self-citationA Comprehensive Review on Automated Grading Systems in STEM Using AI Techniques
Tan, L. Y., Hu, S., Yeo, D. J. S., & Cheong, K. H. (2025) · Mathematics · doi:10.3390/math13172828 · Cited by 28 · see citing works
Gao, Y., Zhai, X.-M., Li, M., Lee, G., & Liu, X.-X. (2025) · Journal of Research in Science Teaching · doi:10.1002/tea.70009 · Cited by 22 · see citing works
Self-citationHan, C., Lu, X., & Fan, Q. (2025) · The Interpreter and Translator Trainer · doi:10.1080/1750399x.2025.2533606 · Cited by 24 · see citing works
Atreja, S., Ashkinaze, J., Li, L., Mendelsohn, J., & Hemphill, L. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35807 · Cited by 34 · see citing works
NERIF: GPT-4V for Automatic Scoring of Drawn Models
Lee, G.-G., & Zhaı, X. (2025) · Journal of Science Education and Technology · doi:10.1007/s10956-025-10262-9 · Cited by 23 · see citing works
Self-citationHarnessing large language models to auto-evaluate the student project reports
Du, H., Jia, Q., Gehringer, E. F., & Wang, X. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100268 · Cited by 20 · see citing works
Evaluation of Transfer Learning and Adaptability in Large Language Models with the GLUE BenchmarkPreprint
Sulaiman, N., & Hamzah, F. (2024) · doi:10.36227/techrxiv.171077989.99407624/v1 · Cited by 20 · see citing works
Fock, A., & Siller, H.-S. (2025) · International Journal of STEM Education · doi:10.1186/s40594-025-00589-5 · Cited by 21 · see citing works
Demystifying large language models in second language development research
Cong, Y. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101700 · Cited by 21 · see citing works
Lee, U., Kim, Y., Lee, S., Park, J., Mun, J., Lee, E., Kim, H., Lim, C., & Yoo, Y. J. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00448-4 · Cited by 20 · see citing works
Narreddy, C., Joordens, S., & Prompiengchai, S. (2025) · Trends in Higher Education · doi:10.3390/higheredu4040065 · Cited by 21 · see citing works
Martin, P. P., & Graulich, N. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100265 · Cited by 19 · see citing works
Research on Intelligent Grading of Physics Problems Based on Large Language Models
Wei, Y., Zhang, R., Zhang, J., Qi, D., & Cui, W. (2025) · Education Sciences · doi:10.3390/educsci15020116 · Cited by 18 · see citing works
LLM-CDM: A Large Language Model Enhanced Cognitive Diagnosis for Intelligent Education
Chen, X., Zhang, J., Zhou, T., & Zhang, F. (2025) · IEEE Access · doi:10.1109/access.2025.3549309 · Cited by 15 · see citing works
Jung, J. Y., Tyack, L., & von Davier, M. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100375 · Cited by 14 · see citing works
Sui, C., Chang, C., & Yen, M.-H. (2025) · Journal of Science Education and Technology · doi:10.1007/s10956-025-10257-6 · Cited by 13 · see citing works
Automatic feedback on physics tasks using open-source generative artificial intelligence
Meyer, A., Bleckmann, T., & Friege, G. (2025) · International Journal of Science Education · doi:10.1080/09500693.2025.2499220 · Cited by 12 · see citing works
Pan, Y., & Nehm, R. H. (2025) · Education Sciences · doi:10.3390/educsci15060676 · Cited by 12 · see citing works
Assessing and Analyzing the Correctness of GitHub Copilot's Code Suggestions
Mo, R., Wang, D., Zhan, W., Jiang, Y., Wang, Y., Zhao, Y., Li, Z., & Ma, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715108 · Cited by 11 · see citing works
Jiang, P. (2025) · Journal of Engineering Design · doi:10.1080/09544828.2025.2574209 · Cited by 11 · see citing works
Nguyen, H., & Park, S. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706480 · Cited by 10 · see citing works
Evaluating a Chatbot as a Companion for Patients With Breast Cancer: Collaborative Pilot Study
Boie, S. D., Glastetter, E., Lux, M. P., Balzer, F., Kalle, C. V., Lenz, C., & Müller, U. (2025) · JMIR Cancer · doi:10.2196/68426 · Cited by 11 · see citing works
Cordero, J., & Cordero-Castillo, A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-85930-4_23 · Cited by 12 · see citing works
Benchmarking Large Language Models on Homework Assessment in Circuit Analysis
Chen, L., Qin, Z., Guo, Y., Rohde, J. A., & Zhang, Y. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00501-w · Cited by 9 · see citing works
Vision-Language Models for Highway Roadside Safety Management: A Comparative Study
Sadeghi, M., Mansouri, A., & Erfani, A. (2025) · Journal of Management in Engineering · doi:10.1061/jmenea.meeng-7141 · Cited by 11 · see citing works
Sawaki, Y., Ishii, Y., Yamada, H., & Tokunaga, T. (2025) · Language Testing · doi:10.1177/02655322251349217 · Cited by 12 · see citing works
Automatic Prompt Engineering for Automatic Scoring
Xue, M.-F., Liu, Y., Xiao, X.-Y., & Wilson, M. (2025) · Journal of Educational Measurement · doi:10.1111/jedm.70002 · Cited by 7 · see citing works
On the Consistency of Automatic Scoring with Large Language Models
Xue, M., Xiao, X., Liu, Y., & Wilson, M. (2026) · Educational and Psychological Measurement · doi:10.1177/00131644261418138 · Cited by 7 · see citing works
Does Multiple Choice Have a Future in the Age of Generative AI? A Posttest-only RCT
Thomas, D. R., Borchers, C., Kakarla, S., Lin, J., Bhushan, S., Guo, B., Gatz, E. E., & Koedinger, K. R. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706530 · Cited by 7 · see citing works
Automated Genre-Aware Article Scoring and Feedback Using Large Language Models
Wang, C., Dong, Y., Zhang, Z., Wang, R., Wang, S., & Chen, J. (2024) · International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) · doi:10.1109/icbaie63306.2024.11116923 · Cited by 10 · see citing works
Enhancing Essay Scoring: An Analytical and Holistic Approach With Few-Shot Transformer-Based Models
Amin, T., Rehman, Z. U., Aadil, F., Awan, K. M., & Lim, S. (2025) · IEEE Access · doi:10.1109/access.2025.3530272 · Cited by 6 · see citing works
Jere, S. (2025) · Eurasia Journal of Mathematics Science and Technology Education · doi:10.29333/ejmste/15932 · Cited by 8 · see citing works
Generative AI in assessing written responses of geography exams: challenges and potential
Jauhiainen, J. S., Guerra, A. G., Nylén, T., & Mäki, S. (2025) · Journal of Geography in Higher Education · doi:10.1080/03098265.2025.2593484 · Cited by 5 · see citing works
Generative artificial intelligence (AI): a key innovation or just hype in primary care settings?
Rahmanti, A. R., Iqbal, U., Reddy, S., Gao, X. W., Nguyen, H. X., & Li, Y. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2024-101367 · Cited by 5 · see citing works
Reflective prompt engineering: a new strategy for automated short answer scoring in biology
Ariely, M., Salman, A., Yarden, A., & Alexandron, G. (2025) · International Journal of Science Education · doi:10.1080/09500693.2025.2523571 · Cited by 4 · see citing works
Frohn, S., Burleigh, T. J., & Chen, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98465-5_6 · Cited by 4 · see citing works
Challenges and feasibility of multimodal LLMs in ER diagram evaluation
Rahmanian, M., Sami, A., & Yu, Y. (2025) · Cogent Education · doi:10.1080/2331186x.2025.2590901 · Cited by 4 · see citing works
Exploring potential of large language models for automated essay scoring in education
Mughal, N., Imran, A. S., Daudpota, S. M., Kastrati, Z., & Noor, W. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01002-y · Cited by 6 · see citing works
Sáez, Y., Garcia, L. M., Mochón, A., & Isasi, P. (2026) · Scientific Reports · doi:10.1038/s41598-026-48656-3 · Cited by 6 · see citing works
Song, D., Lee, W., & Jiao, H. (2026) · International Journal of Testing · doi:10.1080/15305058.2026.2684584 · Cited by 7 · see citing works
Hybrid ANN-Based and Text Similarity Method for Automatic Short-Answer Grading in Polish
Saad, M. B., Jackowska-Strumiłło, L., & Bieniecki, W. (2025) · Applied Sciences · doi:10.3390/app15031605 · Cited by 4 · see citing works
Application possibilities of artificial intelligence in the field of transport and logistics
Kostovčík, M., Fedorko, G., Mikušová, N., Molnár, V., Neradilová, H., & Ďuriška, M. (2025) · Transportation Research Procedia · doi:10.1016/j.trpro.2025.04.109 · Cited by 3 · see citing works
The design of prompts driven by Chain of Thought in multicultural teaching
Chen, Y., Li, Y., & Yang, J. (2024) · International Conference on Signal Processing · doi:10.1109/icsp62129.2024.10846743 · Cited by 4 · see citing works
Chen, X., Chen, Z., & Cheng, S. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00041-2 · Cited by 6 · see citing works
Survey and Analysis for the Challenges in Computer Science to the Automation of Grading Systems
Lu, J., Balasubramanian, B. K., Joy, M., & Xu, Q. (2025) · ACM Computing Surveys · doi:10.1145/3748521 · Cited by 5 · see citing works
Evaluation of a Node-based Automatic Short Answer Tool “NodeGrade”
Fischer, D. V., Haug, J., Schoppel, P., Abke, J., Becker, M., & Hagel, G. (2025) · European Conference on Software Engineering Education (ECSEE) · doi:10.1145/3723010.3723021 · Cited by 3 · see citing works
DentEval: Fine-tuning-Free Expert-Aligned Assessment in Dental Education via LLM Agents
Deng, X., Miletić, V., Trinh, E., Gao, J., Xu, C., & Liu, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04971-1_14 · Cited by 3 · see citing works
Binsar, F., & Sasmoko, S. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.08.291 · Cited by 5 · see citing works
Li, K., Qian, C., & Yang, X. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12851-4 · Cited by 4 · see citing works
Tang, S., Lei, C., & Wang, H. (2024) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale62452.2024.10834357 · Cited by 2 · see citing works
Chakraborty, S., Faouzi, N. E., & Furno, A. (2026) · Applied Sciences · doi:10.3390/app16104716 · Cited by 2 · see citing works
Discovering technology-aided possibilities for automatic analysis of science teacher questions
Lehesvuori, S., Urrutia, F., Heilala, V., Araya, R., & Hämäläinen, R. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10545-3 · Cited by 2 · see citing works
Neural Network with Fine-Tuned BERT for IELTS Writing Evaluation
Ilyas, D., Mimenbayeva, A., & Kadirbayeva, A. (2025) · IEEE International Conference on Smart Information Systems and Technologies (SIST) · doi:10.1109/sist61657.2025.11139356 · Cited by 2 · see citing works
Leveraging Large Language Models for Evaluating Education Teaching Content Quality
Qi, Z., & Zhang, R. (2025) · International Conference on Distance Education and Learning (ICDEL) · doi:10.1109/icdel65868.2025.11193448 · Cited by 2 · see citing works
GenScore: Agent-Based Short-Answer Question Generation and Scoring in Software Engineering Courses
Duong, T. N. B., & Shar, L. K. (2025) · IEEE Access · doi:10.1109/access.2025.3647111 · Cited by 3 · see citing works
Multi-Dimensional Evaluation of Auto-Generated Chain-of-Thought Traces in Reasoning Models
Becerra-Monsalve, L. F., Sanchez-Torres, G., & Bedoya, J. B. B. (2026) · Applied Informatics · doi:10.3390/ai7010035 · Cited by 3 · see citing works
Tian, Y., Huynh, L., Christhilf, K., Chakraborty, S., Watanabe, M., Arner, T., & McNamara, D. S. (2026) · Electronics · doi:10.3390/electronics15061209 · Cited by 3 · see citing works
Norgaila, E., Daniela, L., & Kalniņa, D. (2026) · Technology Knowledge and Learning · doi:10.1007/s10758-026-09979-2 · Cited by 2 · see citing works
Evaluating large language models for rubric-based essay grading in an undergraduate biology course
Naidu, M., Montaquila, N. S., Roa, J. P., & Achilli, T.-M. (2026) · Journal of Microbiology and Biology Education · doi:10.1128/jmbe.00095-26 · Cited by 2 · see citing works
A Student Emotions Recognition Based Online Texts with Large Language Models for Learning Prediction
Fan, G., Zhang, S., Cheng, S., & Yang, D. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822338 · Cited by 2 · see citing works
Mazzone, S. B., Forden, J., & Brylow, D. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892816 · Cited by 2 · see citing works
Aryadoust, V., & Jia, Y. (2025) · Springer eBooks · doi:10.1007/978-3-031-51447-0_201-1 · Cited by 2 · see citing works
SENEM-AI: Leveraging LLMs for student behavior simulation in virtual learning environments
Pentangelo, V., Turco, L., Lambiase, S., Gravino, C., & Palomba, F. (2025) · SoftwareX · doi:10.1016/j.softx.2025.102278 · Cited by 2 · see citing works
Exploring Large Language Models for Automated Essay Grading in Finance Domain
Malik, G., Çevik, M., & Lee, S. (2024) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon62161.2024.10838105 · Cited by 1 · see citing works
Hwang, W., Nguyen, Q.-V., & Sun, M.-T. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129939 · Cited by 1 · see citing works
Jin, H., Lima, C., & Wang, L. (2025) · Educational Measurement Issues and Practice · doi:10.1111/emip.70003 · Cited by 1 · see citing works
Stark, C., Stanzilis, K., Plauche, S., & Palacios, A. F. (2025) · Families in Society The Journal of Contemporary Social Services · doi:10.1177/10443894251378612 · Cited by 1 · see citing works
Radišić, R., Popov, S., & Ralević, N. (2025) · Mathematics · doi:10.3390/math14010137 · Cited by 1 · see citing works
Çiçek, V., Zhao, L., Tur, Y., Öz, A., Kılıç, Ş., Durak, G., Şaylık, F., Hayıroğlu, M. İ., Çınar, T., & Bağcı, U. (2026) · Medical Sciences · doi:10.3390/medsci14010033 · Cited by 1 · see citing works
Rachmatullah, A., Tayde, S., Alozie, N. M., Park, S., & Capan, S. (2026) · Disciplinary and Interdisciplinary Science Education Research · doi:10.1186/s43031-025-00151-x · Cited by 1 · see citing works
Benchmarking Large Language Models for Solving Ordinary Differential Equations
Samanjata, D., Santin, R. V., Marcacini, R. M., & Rezende, S. O. (2025) · International Conference on Soft Computing & Machine Intelligence (ISCMI) · doi:10.1109/iscmi67495.2025.11358441 · Cited by 1 · see citing works
Meyer, A., & Friege, G. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1759878 · Cited by 1 · see citing works
- Authors
- Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X.
- Affiliations
- University of Georgia (US)
- Venue
- Computers and Education: Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2024-06
- Volume
- 6
- Pages
- 100213
- ISSN
- 2666-920X (ISSN-L)
- OpenAlex
- W4392202731
- Cited by
- 186
- References
- 58
- Citations by year
- 2026: 80 · 2025: 73 · 2024: 33
- FWCI
- 69.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Computers and Education Artificial Intelligence (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2101104)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- automatic scoring, large language models, chain-of-thought prompting, GPT-4, GPT-3.5, prompt engineering, few-shot learning, zero-shot learning, rubrics, science assessment, student response, explainable AI, greedy sampling, nucleus sampling, ensemble voting, proficiency level
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
