CoChat
Sign Up
Cited by 188Artificial Intelligence in EducationEducational Technology and AssessmentIntelligent Tutoring Systems and Adaptive Learning

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.

Applying large language models and chain-of-thought for automatic scoring

Open access
Lee, Gyeong-Geon, Latif, Ehsan, Wu, Xuansheng, Liu, Ninghao, Zhai, Xiaoming·2024·Computers and Education: Artificial Intelligence
DOI 10.1016/j.caeai.2024.100213
Open this paper in CoChat — free account

Abstract

This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics, developed based on a novel approach, WRVRT (prompt writing, reviewing, validating, revising, and testing). Results indicated that few-shot (acc = 0.67) outperformed zero-shot learning (acc = 0.60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = 0.60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT -3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches. This study also demonstrates the potential of LLMs in facilitating explainable and interpretable automatic scoring, emphasizing that CoT enhances accuracy and transparency, particularly when used with item stem and scoring rubrics.

Overview

Integrity signals

Registry

Crossref · 10.1016/j.caeai.2024.100213

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 58 references

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X.
Affiliations
University of Georgia (US)
ORCID
Venue
Computers and Education: Artificial Intelligence
Publisher
Elsevier BV
Published
2024-06
Volume
6
Pages
100213
ISSN
2666-920X (ISSN-L)
DOI
10.1016/j.caeai.2024.100213
OpenAlex
W4392202731
Cited by
186
References
58
Citations by year
2026: 80 · 2025: 73 · 2024: 33
FWCI
69.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Computers and Education Artificial Intelligence (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
National Science Foundation (2101104)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
automatic scoring, large language models, chain-of-thought prompting, GPT-4, GPT-3.5, prompt engineering, few-shot learning, zero-shot learning, rubrics, science assessment, student response, explainable AI, greedy sampling, nucleus sampling, ensemble voting, proficiency level

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review