CoChat
Sign Up
Cited by 44Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningEducational Technology and Assessment

Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability To Mark Short Answer Questions in K-12 Education

Open access
Henkel, Owen, Hills, Libby, Boxer, Adam, Roberts, Bill, Levonian, Zach·2024·Proceedings of the Eleventh ACM Conference on Learning @ Scale
DOI 10.1145/3657604.3664693
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper presents reports on a series of experiments with a novel dataset evaluating how well Large Language Models (LLMs) can mark (i.e. grade) open text responses to short answer questions, Specifically, we explore how well different combinations of GPT version and prompt engineering strategies performed at marking real student answers to short answer across different domain areas (Science and History) and grade-levels (spanning ages 5-16) using a new, never-used-before dataset from Carousel, a quizzing platform. We found that GPT-4, with basic few-shot prompting performed well (Kappa, 0.70) and, importantly, very close to human-level performance (0.75). This research builds on prior findings that GPT-4 could reliably score short answer reading comprehension questions at a performance-level very close to that of expert human raters. The proximity to human-level performance, across a variety of subjects and grade levels suggests that LLMs could be a valuable tool for supporting low-stakes formative assessment tasks in K-12 education and has important implications for real-world education delivery.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3657604.3664693

Cites

19 of 23 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior studies have demonstrated that LLMs can reach high agreement with human annotators when grading correctness of responses to short-answer questions on existing datasets.

    “Studies (Thakur et al., 2025;Badshah & Sajjad, 2024;Henkel et al., 2024;Badshah et al., 2025) have demonstrated that LLMs can reach high agreement with human annotators when grading the correctness of responses to short-answer questions on existing datasets.”

    From No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding (2025) · cites this paper as Henkel et al., 2024

    The Introduction reports that GPT-4 achieved close to human-level agreement when classifying short-answer responses as correct or incorrect (κ = 0.70 versus 0.75). However, §2.1 says the authors created a new dataset from Carousel student responses, so this paper does not support the claim’s characterization of its evaluation as being on an existing dataset. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Bommasani, R. et al. 2022. On the Opportunities and Risks of Foundation Models. arXiv.
  2. Kuzman, T., Mozetič, I. and Ljubešić, N. 2023. ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification. arXiv.
  3. Zhao, S., Li, B., Reed, C., Xu, P. and Keutzer, K. 2020. Multi-source Domain Adaptation in the Deep Learning Era: A Systematic Survey. arXiv.

Cited by

Sorted by citations

Related

Authors
Henkel, O., Hills, L., Boxer, A., Roberts, B., & Levonian, Z.
Affiliations
  • University of Oxford (GB)
  • Jacobs Foundation (CH)
  • Carousel Learning, London, United Kingdom
  • Legible Labs, New York, USA
  • Digital Harbor Foundation, Baltimore, USA
ORCID
Venue
Proceedings of the Eleventh ACM Conference on Learning @ Scale
Publisher
ACM
Published
2024-07-09
Pages
300-304
DOI
10.1145/3657604.3664693
OpenAlex
W4400668042
Cited by
42
References
19
Citations by year
2026: 15 · 2025: 26 · 2024: 1
FWCI
15.99
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Learning @ Scale (L@S) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
Keywords
large language models, automated grading, short-answer questions, K-12 education, GPT-4, prompt engineering, few-shot prompting, student response, formative assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review