CoChat
Sign Up
Cited by 123Artificial Intelligence in EducationStudent Assessment and FeedbackEFL/ESL Teaching and Learning

Utilizing large language models for EFL essay grading: An examination of reliability and validity in rubric‐based assessments

Open access
Yavuz, Fatih, Çelik, Özgür, Yavaş Çelik, Gamze·2025·British Journal of Educational Technology
DOI 10.1111/bjet.13494
Open this paper in CoChat — free account

Abstract

This study investigates the validity and reliability of generative large language models (LLMs), specifically ChatGPT and Google's Bard, in grading student essays in higher education based on an analytical grading rubric. A total of 15 experienced English as a foreign language (EFL) instructors and two LLMs were asked to evaluate three student essays of varying quality. The grading scale comprised five domains: grammar, content, organization, style & expression and mechanics. The results revealed that fine‐tuned ChatGPT model demonstrated a very high level of reliability with an intraclass correlation (ICC) score of 0.972, Default ChatGPT model exhibited an ICC score of 0.947 and Bard showed a substantial level of reliability with an ICC score of 0.919. Additionally, a significant overlap was observed in certain domains when comparing the grades assigned by LLMs and human raters. In conclusion, the findings suggest that while LLMs demonstrated a notable consistency and potential for grading competency, further fine‐tuning and adjustment are needed for a more nuanced understanding of non‐objective essay criteria. The study not only offers insights into the potential use of LLMs in grading student essays but also highlights the need for continued development and research. Practitioner notes: What is already known about this topic Large language models (LLMs), such as OpenAI's ChatGPT and Google's Bard, are known for their ability to generate text that mimics human‐like conversation and writing. LLMs can perform various tasks, including essay grading. Intraclass correlation (ICC) is a statistical measure used to assess the reliability of ratings given by different raters (in this case, EFL instructors and LLMs). What this paper adds The study makes a unique contribution by directly comparing the grading performance of expert EFL instructors with two LLMs—ChatGPT and Bard—using an analytical grading scale. It provides robust empirical evidence showing high reliability of LLMs in grading essays, supported by high ICC scores. It specifically highlights that the overall efficacy of LLMs extends to certain domains of essay grading. Implications for practice and/or policy The findings open up potential new avenues for utilizing LLMs in academic settings, particularly for grading student essays, thereby possibly alleviating workload of educators. The paper's insistence on the need for further fine‐tuning of LLMs underlines the continual interplay between technological advancement and its practical applications. The results lay down a footprint for future research in advancing the use of AI in essay grading.

Integrity signals

Registry

Crossref · 10.1111/bjet.13494

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The rubric-score consistency observed for AI feedback is best read not as an inherent property of GPT-4o but as an outcome of the prompt configuration.

    “The rubric-score consistency we observed for AI feedback is therefore best read not as an inherent property of GPT-4o but as an outcome of this prompt configuration, consistent with experimental evidence that detailed prompts combined with low temperature settings yield near-perfect interreplicate consistency in LLM rubric grading [66].”

    From AI-Generated Versus Human Supervisor Feedback on Medical Students’ Clinical Clerkship Logs: Cross-Sectional Convergent Mixed Methods Study (2026) · cites this paper as [66]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Yavuz, F., Çelik, Ö., & Yavaş Çelik, G.
Affiliations
  • Mudanya Üniversitesi (TR)
  • Balıkesir University (TR)
  • School of Foreign Languages Balıkesir University Balıkesir Turkey
Corresponding author
Yavuz, Fatih
ORCID
Venue
British Journal of Educational Technology
Publisher
Wiley
Published
2024-06-04
Volume
56
Issue
1
Pages
150-166
ISSN
  • 0007-1013 (ISSN-L)
  • 1467-8535
DOI
10.1111/bjet.13494
OpenAlex
W4399334961
Cited by
121
References
25
Citations by year
2026: 73 · 2025: 44 · 2024: 4
FWCI
45.50
Type
journal article
Language
en
Open access
Licence
Locations
British Journal of Educational Technology (publishedVersion, cc-by-nc-nd)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Social Sciences › Education › Student Assessment and Feedback
  • Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
Keywords
automated essay scoring, ChatGPT, Google Bard, rubric-based assessment, rating reliability, fine-tuning, higher education, instructor workload

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review