The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Applying large language models and chain-of-thought for automatic scoring
Open accessAbstract
This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics, developed based on a novel approach, WRVRT (prompt writing, reviewing, validating, revising, and testing). Results indicated that few-shot (acc = 0.67) outperformed zero-shot learning (acc = 0.60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = 0.60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT -3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches. This study also demonstrates the potential of LLMs in facilitating explainable and interpretable automatic scoring, emphasizing that CoT enhances accuracy and transparency, particularly when used with item stem and scoring rubrics.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite considerable technical interventions aimed at reducing hallucinations-such as chain-of-thought and retrieval-augmented generation prompting (Gao et al. 2023;Lee et al. 2024)-hallucinations remain a persistent limitation of LLMs.”
From Bridging risk and retention: risk coping with privacy and hallucination in AI (2026) · cites this paper as Lee et al. 2024
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
BAİDOO-ANU, D., & OWUSU ANSAH, L. (2023) · Journal of AI · doi:10.61969/jai.1337500 · Cited by 3,939
Baidoo-Anu, D., & Owusu Ansah, L. (2023) · doi:10.2139/ssrn.4337484 · Cited by 3,004
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Techniques for interpretable machine learning
Du, M., Liu, N., & Hu, X. (2019) · Communications of the ACM · doi:10.1145/3359786 · Cited by 1,268
Self-citationTree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080
Explainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556
An automated essay scoring systems: a systematic literature review
Ramesh, D., & Sanampudi, S. K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10068-2 · Cited by 550
Explainable AI Methods - A Brief Overview
Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_2 · Cited by 518
Contrastive Decoding: Open-ended Text Generation as Optimization
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J. M., Hashimoto, T., Zettlemoyer, L., & Lewis, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.687 · Cited by 408
C-rater: Automated Scoring of Short-Answer Questions
Leacock, C., & Chodorow, M. S. (2003) · Computers and the Humanities · doi:10.1023/a:1025779619903 · Cited by 384
MathPrompter: Mathematical Reasoning using Large Language Models
Imani, S., Du, L., & Shrivastava, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.4 · Cited by 382
Learning to work with the black box: Pedagogy for a world with artificial intelligence
Bearman, M., & Ajjawi, R. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13337 · Cited by 346
Applying machine learning in science assessment: A systematic review
Zhai, X., Yin, Y., Pellegrino, J. W., Haudek, K. C., & Shi, L. (2020) · Studies in Science Education · doi:10.1080/03057267.2020.1735757 · Cited by 257
Fine-tuning ChatGPT for automatic scoring
Latif, E., & Zhaı, X. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100210 · Cited by 244
Self-citationA Review on Word Embedding Techniques for Text Classification
Birunda, S. S., & Devi, R. K. (2021) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-15-9651-3_23 · Cited by 224
Lee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., & Kim, H. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12249-8 · Cited by 217
Organisciak, P., Acar, S., Dumas, D., & Berthiaume, K. (2023) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2023.101356 · Cited by 202
Chatgpt for next generation science learning
Zhai, X. (2023) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/3589649 · Cited by 178
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. J. L., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.82 · Cited by 176
A Contrastive Framework for Neural Text Generation
Su, Y., Lan, T., Wang, Y., Yogatama, D., Kong, L., & Collier, N. (2022) · neural information processing systems · doi:10.52202/068431-1566 · Cited by 167
Unpacking the "Black Box" of AI in EducationPreprint
Gillani, N., Eynon, R., Chiabaut, C., & Finkel, K. (2022) · arXiv (Cornell University) · doi:10.30191/ets.202301_26(1).0008 · Cited by 165
Nehm, R. H., Ha, M., & Mayfield, E. (2011) · Journal of Science Education and Technology · doi:10.1007/s10956-011-9300-9 · Cited by 150
From substitution to redefinition: A framework of machine learning‐based science assessment
Zhaı, X., Haudek, K. C., Shi, L., Nehm, R. H., & Urban‐Lurain, M. (2020) · Journal of Research in Science Teaching · doi:10.1002/tea.21658 · Cited by 145
Meister, C., Pimentel, T., Wiher, G., & Cotterell, R. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00536 · Cited by 137
Language models and Automated Essay ScoringPreprint
Rodriguez, P. U., Jafari, A., & Ormerod, C. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09482 · Cited by 134
Automated essay scoring with string kernels and word embeddings
Cozma, M., Butnaru, A. M., & Ionescu, R. T. (2018) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p18-2080 · Cited by 127
Applying machine learning to automatically assess scientific models
Zhaı, X., He, P., & Krajcik, J. S. (2022) · Journal of Research in Science Teaching · doi:10.1002/tea.21773 · Cited by 127
Hahn, M. G., Navarro, S. M. B., De La Fuente Valentin, L., & Burgos, D. (2021) · IEEE Access · doi:10.1109/access.2021.3100890 · Cited by 111
Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to TransformersPreprint
Haller, S., Aldea, A., Seifert, C., & Strisciuglio, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.03503 · Cited by 83
A Contrastive Framework for Neural Text GenerationPreprint
Su, Y., Lan, T., Wang, Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.06417 · Cited by 83
Context Matters: A Strategy to Pre-train Language Model for Science Education
Liu, Z., He, X., Liu, L., Liu, T., & Zhaı, X. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-36336-8_103 · Cited by 81
Jescovitch, L. N., Scott, E. E., Cerchiara, J. A., MERRILL, J. E., Urban‐Lurain, M., Doherty, J. H., & Haudek, K. C. (2020) · Journal of Science Education and Technology · doi:10.1007/s10956-020-09858-0 · Cited by 78
Bewersdorff, A., Seßler, K., Baur, A., Kasneci, E., & Nerdel, C. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100177 · Cited by 76
A review of research on machine learning in educational technology
Korkmaz, C., & Correia, A. (2019) · Educational Media International · doi:10.1080/09523987.2019.1669875 · Cited by 73
Truncation Sampling as Language Model Desmoothing
Hewitt, J., Manning, C. D., & Liang, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.249 · Cited by 70
A Theoretical Analysis of the Repetition Problem in Text Generation
Fu, Z., Lam, W. P., So, A. M., & Shi, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17520 · Cited by 68
Binding Language Models in Symbolic Languages
Cheng, Z., Xie, T., Shi, P., Li, C., Nadkarni, R., Hu, Y., Xiong, C., Radev, D., Ostendorf, M., Zettlemoyer, L., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02875 · Cited by 53
Assessing Argumentation Using Machine Learning and Cognitive Diagnostic Modeling
Zhaı, X., Haudek, K. C., & Ma, W. (2022) · Research in Science Education · doi:10.1007/s11165-022-10062-w · Cited by 51
Using automated analysis to assess middle school students' competence with scientific argumentation
Wilson, C. D., Haudek, K. C., Osborne, J. F., Bracey, Z. B., Cheuk, T., Donovan, B., Stuhlsatz, M. A. M., Santiago, M. M., & Zhaı, X. (2023) · Journal of Research in Science Teaching · doi:10.1002/tea.21864 · Cited by 49
Uses of Artificial Intelligence in STEM Education
(2024) · doi:10.1093/oso/9780198882077.001.0001 · Cited by 49
Gemini Pro Defeated by GPT-4V: Evidence from EducationPreprint
Lee, G.-G., Latif, E., Shi, L., & Zhaı, X. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.17393.25448/1 · Cited by 48
Self-citationMathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
Shen, J. T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., Graff, B., & Lee, D. (2021) · arXiv · doi:10.48550/arxiv.2106.07340 · Cited by 47
Self-citationZhaı, X., Haudek, K. C., Stuhlsatz, M. A. M., & Wilson, C. D. (2020) · Studies In Educational Evaluation · doi:10.1016/j.stueduc.2020.100916 · Cited by 40
Gemini Pro Defeated by GPT-4V: Evidence from EducationPreprint
Lee, G.-G., Latif, E., Shi, L.-H., & Zhai, X. (2023) · arXiv.org · doi:10.48550/arxiv.2401.08660 · Cited by 40
Self-citationChatGPT and AI: The Game Changer for Education
Zhai, X. (2023) · doi:10.13140/rg.2.2.31107.37923 · Cited by 40
Self-citationWu, X., He, X., Liu, T., Liu, N., & Zhaı, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36272-9_33 · Cited by 39
Self-citationAdvancing automatic guidance in virtual science inquiry: from ease of use to personalization
Zhaı, X. (2021) · Educational Technology Research and Development · doi:10.1007/s11423-020-09917-8 · Cited by 25
Muhamedyev, R. I., Yakunin, K., Kuchin, Y. I., Сымагулов, А., Buldybayev, T., Murzakhmetov, S., & Abdurazakov, A. (2020) · Cogent Engineering · doi:10.1080/23311916.2020.1769349 · Cited by 22
G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in GermanPreprint
Latif, E., Lee, G.-G., Neumann, K., Kastorff, T., & Zhaı, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.06584 · Cited by 18
Self-citationChen, Z., Zhou, Q., Shen, Y., Hong, Y., Zhang, H., & Gan, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.05226 · Cited by 8
Cited by
Sorted by citations
Using ChatGPT for Science Learning: A Study on Pre-service Teachers' Lesson Planning
Lee, G.-G., & Zhaı, X. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3401457 · Cited by 264
Self-citationZhai, X. (2025) · Journal of Science Education and Technology · doi:10.1007/s10956-024-10174-0 · Cited by 184
Prompt Engineering in Education: A Systematic Review of Approaches and Educational Applications
Qian, Y. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251365189 · Cited by 134
Flodén, J. (2024) · British Educational Research Journal · doi:10.1002/berj.4069 · Cited by 124
Seßler, K., Fürstenberg, M., Bühler, B., & Kasneci, E. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706527 · Cited by 99
Metadata
- Authors
- Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X.
- Affiliations
- University of Georgia (US)
- Venue
- Computers and Education: Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2024-06
- Volume
- 6
- Pages
- 100213
- ISSN
- 2666-920X (ISSN-L)
- OpenAlex
- W4392202731
- Cited by
- 186
- References
- 58
- Citations by year
- 2026: 80 · 2025: 73 · 2024: 33
- FWCI
- 69.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Computers and Education Artificial Intelligence (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2101104)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- automatic scoring, large language models, chain-of-thought prompting, GPT-4, GPT-3.5, prompt engineering, few-shot learning, zero-shot learning, rubrics, science assessment, student response, explainable AI, greedy sampling, nucleus sampling, ensemble voting, proficiency level
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
