Can AI Provide Useful Holistic Essay Scoring?
Open accessAbstract
Researchers have sought for decades to automate holistic essay scoring. Over the years, these programs have improved significantly. However, accuracy requires significant amounts of training on human-scored texts—reducing the expediency and usefulness of such programs for routine uses by teachers across the nation on non-standardized prompts. This study analyzes the output of multiple versions of ChatGPT scoring of secondary student essays from three extant corpora and compares it to quality human ratings. We find that the current iteration of ChatGPT scoring is not statistically significantly different from human scoring; substantial agreement with humans is achievable and may be sufficient for low-stakes, formative assessment purposes. However, as large language models evolve additional research will be needed to continue to assess their aptitude for this task as well as determine whether their proximity to human scoring can be improved through prompting or training.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Essays marked by humans and GenAI are internally consistent, and the mean differences between human-human and AI-human scores are not statistically significant.
“It has been shown that essays marked by humans and GenAI are internally consistent (i.e., human-human and AI-human scores) and that the mean differences between human-human and AI-human scores are not statistically significant [52].”
From Generative AI in Higher Education: Balancing Innovation and Integrity (2025) · cites this paper as [52]
Cites
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332 · see citing works
A cognitive process theory of writing
Flower, L., & Hayes, J. R. (1981) · College Composition & Communication · doi:10.58680/ccc198115885 · Cited by 1,025 · see citing works
Exploring the potential of using an AI language model for automated essay scoring
Mizumoto, A., & Eguchi, M. (2023) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2023.100050 · Cited by 647 · see citing works
A Revised Writer(s)-Within-Community Model of Writing
Graham, S. (2018) · Educational Psychologist · doi:10.1080/00461520.2018.1481406 · Cited by 599 · see citing works
Self-citationRating the ratings: Assessing the psychometric quality of rating data.
Saal, F. E., Downey, R. G., & Lahey, M. A. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.413 · Cited by 545 · see citing works
Automated writing evaluation: Defining the classroom research agenda
Warschauer, M., & Ware, P. (2006) · Language Teaching Research · doi:10.1191/1362168806lr190oa · Cited by 337 · see citing works
Self-citationEJ Extra: A Snapshot of Writing Instruction in Middle Schools and High Schools [FREE ACCESS]
Applebee, A. N., & Langer, J. A. (2011) · English Journal · doi:10.58680/ej201116413 · Cited by 264 · see citing works
Rater types in writing performance assessments: A classification approach to rater variability
Eckes, T. (2008) · Language Testing · doi:10.1177/0265532207086780 · Cited by 258 · see citing works
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 243 · see citing works
Automated Writing Assessment in the Classroom
Warschauer, M., & Grimes, D. (2008) · Pedagogies: An International Journal · doi:10.1080/15544800701771580 · Cited by 243 · see citing works
Self-citationUtility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation.
Grimes, D., & Warschauer, M. (2010) · The Journal of Technology, Learning and Assessment (Boston College) · Cited by 241 · see citing works
Self-citationDisciplinary Literacy in History: An Exploration of the Historical Nature of Adolescents' Writing
Monte‐Sano, C. (2010) · Journal of the Learning Sciences · doi:10.1080/10508406.2010.481014 · Cited by 182 · see citing works
Using Writing Tasks to Elicit Adolescents’ Historical Reasoning
Monte‐Sano, C., & Paz, S. D. L. (2012) · Journal of Literacy Research · doi:10.1177/1086296x12450445 · Cited by 165 · see citing works
Weigle, S. C. (1999) · Assessing Writing · doi:10.1016/s1075-2935(00)00010-6 · Cited by 157 · see citing works
Handbook of Open, Distance and Digital Education
Zawacki‐Richter, O., & Jung, I. (2023) · Springer eBooks · doi:10.1007/978-981-19-2080-6 · Cited by 150 · see citing works
Proceedings of the 14th International Conference on Educational Data Mining
Hsiao, I., Sahebi, S., Bouchet, F., & Vie, J.-J. (2021) · HAL · Cited by 98 · see citing works
A novel automated essay scoring approach for reliable higher educational assessments
Beseiso, M., Alzubi, O. A., & Rashaideh, H. (2021) · Journal of Computing in Higher Education · doi:10.1007/s12528-021-09283-1 · Cited by 88 · see citing works
The effect of variations in handwriting and print on evaluation of student essays
Klein, J., & Taub, D. (2005) · Assessing Writing · doi:10.1016/j.asw.2005.05.002 · Cited by 53 · see citing works
Student Use of Automated Essay Evaluation Technology During Revision
Moore, N. S., & MacArthur, C. A. (2016) · Journal of Writing Research · doi:10.17239/jowr-2016.08.01.05 · Cited by 51 · see citing works
What Makes a Good History Essay? Assessing Historical Aspects of Argumentative Writing
Monte‐Sano, C. (2012) · Social Education · doi:10.66392/003777212814727273 · Cited by 37 · see citing works
Uto, M., Aomi, I., Tsutsumi, E., & Ueno, M. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3253215 · Cited by 33 · see citing works
Olson, C. B., Woodworth, K. R., Arshan, N., Black, R. W., Chung, H. Q., D'Aoust, C., Dewar, T., Friedrich, L., Godfrey, L., Land, R. E., et al. (2019) · Journal of Educational Psychology · doi:10.1037/edu0000387 · Cited by 30 · see citing works
Chen, D., Hebert, M. A., & Wilson, J. (2022) · American Educational Research Journal · doi:10.3102/00028312221106773 · Cited by 28 · see citing works
The effects of prior computer use on computer-based writing: The 2011 NAEP writing assessment
Tate, T. P., Warschauer, M., & Abedi, J. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.06.001 · Cited by 24 · see citing works
Self-citationEvaluation of ChatGPT Feedback on ELL Writers' Coherence and CohesionPreprint
Yoon, S., Miszoglad, E., & Pierce, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.06505 · Cited by 21 · see citing works
Validating human and automated scoring of essays against “True” scores
Cohen, Y., Levi, E., & Ben-Simon, A. (2018) · Applied Measurement in Education · doi:10.1080/08957347.2018.1464450 · Cited by 20 · see citing works
Learning to Write in Middle School?
Lawrence, J. F., Galloway, E. P., Yim, S., & Lin, A. (2013) · Journal of Adolescent & Adult Literacy · doi:10.1002/jaal.219 · Cited by 19 · see citing works
Automated essay scoring (AES) of constructed responses in nursing examinations: An evaluation
Stephen, T., Gierl, M. J., & King, S. (2021) · Nurse Education in Practice · doi:10.1016/j.nepr.2021.103085 · Cited by 18 · see citing works
Reliability of National Writing Project's Analytic Writing Continuum Assessment System
Bang, H. J. (2013) · eScholarship (California Digital Library) · Cited by 16 · see citing works
LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores
Liu, Y., Moosavi, N. S., & Lin, C. (2023) · arXiv · doi:10.48550/arxiv.2311.09766 · Cited by 13 · see citing works
Automated Feedback Generation for Student Project Reports: A Data-Driven Approach
Jia, Q., Young, M., Xiao, Y., Cui, J., Liu, C., Rashid, P., & Gehringer, E. F. (2022) · Journal of Educational Data Mining · doi:10.5281/zenodo.7304953 · Cited by 11 · see citing works
Ifenthaler, D., & Dikli, S. (2015) · The SAGE Encyclopedia of Educational Technology · doi:10.4135/9781483346397.n35 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Generative AI in Higher Education: Balancing Innovation and Integrity
Francis, N. J., Jones, S., & Smith, D. P. (2025) · British Journal of Biomedical Science · doi:10.3389/bjbs.2024.14048 · Cited by 295 · see citing works
“It has been shown that essays marked by humans and GenAI are internally consistent (i.e., human-human and AI-human scores) and that the mean differences between human-human and AI-human scores are not statistically signi fi cant [52].”
Peer and AI Review + Reflection (PAIRR): A human-centered approach to formative assessment
Sperber, L., MacArthur, M., Minnillo, S., Stillman, N., & Whithaus, C. (2025) · Computers and Composition · doi:10.1016/j.compcom.2025.102921 · Cited by 29 · see citing works
What undergraduate students need to know and actually know about generative AI
Rismanchian, S., Babar, E. T. R., & Doroudi, S. (2026) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2026.100554 · Cited by 5 · see citing works
- Authors
- Tate, T. P., Steiss, J., Bailey, D., Graham, S., Moon, Y., Ritchie, D., Tseng, W., & Warschauer, M.
- Affiliations
- University of California, Irvine (US)
- Arizona State University (US)
- Corresponding author
- Tate, Tamara P.
- ORCID
- Venue
- Computers and Education: Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2024-12
- Volume
- 7
- Pages
- 100255
- ISSN
- 2666-920X (ISSN-L)
- OpenAlex
- W4399619336
- Semantic Scholar
- 3266ff7c9e03020d6978776d66c2088f75588b26
- Cited by
- 60
- References
- 34
- Citations by year
- 2026: 29 · 2025: 26 · 2024: 5
- FWCI
- 22.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Computers and Education Artificial Intelligence (publishedVersion, cc-by-nc)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by-nc)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Funders
- U.S. Department of Education (R305C190007)
- University of California, Irvine (CA 92697-5500)
- National Science Foundation (23152984)
- Institute of Education Sciences
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Social Sciences › Education › Student Assessment and Feedback
- Keywords
- automated essay scoring, holistic scoring, ChatGPT, large language models, formative assessment, low-stakes assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
