CoChat

A large-scale comparison of human-written versus ChatGPT-generated essays

Open access

The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index; 2 of the 15 sources its claims cite were checked for support.

Abstract

This study compares human-written and ChatGPT-generated argumentative essays and reports differing linguistic characteristics.

Overview

CoChat Summary

TL;DR

In a large-scale comparison, ChatGPT-generated argumentative essays were rated higher than human-written essays across all seven criteria, with ChatGPT-4 best overall (M=4.68 vs ChatGPT-3 4.36 vs humans 3.69 on a 0–6 scale). The models also differed linguistically from students, e.g., more nominalizations and fewer modals/epistemic markers.

Key findings

  • ChatGPT-4 had the highest mean rating for every criterion; overall means were 4.68 for ChatGPT-4, 4.36 for ChatGPT-3, and 3.69 for humans. Human vs. ChatGPT-3 effect sizes ranged from 0.52 to 1.15; human vs. ChatGPT-4 from 0.88 to 1.43. ChatGPT-4 vs. ChatGPT-3 differences were significant only for logic and composition, vocabulary and text linking, and complexity, with effect sizes 0.45–0.5.
  • Inter-rater agreement was consistently excellent (Cronbach’s α 0.95–0.97) except language mastery, which was good (α=0.89).
  • Linguistic markers differed: lexical diversity means were humans 95.72, ChatGPT-3 75.68, ChatGPT-4 108.91; nominalizations 1.06, 1.56, 1.73; modals 10.84, 8.97, 6.12; epistemic markers 0.06, 0.02, 0.00; discourse markers 0.57, 0.52, 0.36. Sentence complexity (depth) means were 5.72, 6.18, 5.94; clause complexity 1.81, 2.31, 2.08.
  • Complexity ratings correlated positively with sentence complexity and nominalizations, vocabulary ratings with lexical diversity, and logic ratings negatively with discourse markers; all correlations were significant but weak.

Methods

  • Data: 90 argumentative essay topics from Stab and Gurevych’s corpus; human essays came from Essay Forum and were written by non-native high-school students likely in grades 11–13. ChatGPT-3 and ChatGPT-4 browser versions were prompted on 22 March 2023 with “Write an essay with about 200 words on ‘[topic]’” in a zero-shot setting, yielding 270 essays total (90 per source).
  • Annotation: 111 teachers completed the questionnaire, producing 658 ratings for 270 essays; each participant was shown six randomly selected essays without source information. Essays were rated on seven criteria from Lower Saxony guidelines using a seven-point Likert scale (0–6), plus self-assessed English skills and rating confidence.
  • Analysis: within-subjects design; Cronbach’s α for inter-rater reliability; two-sided Wilcoxon rank-sum tests with Holm-Bonferroni correction; Cohen’s d; Pearson correlations. Computational linguistic measures included MTLD lexical diversity, spaCy dependency depth, clause counts, nominalization suffixes, modals, epistemic markers, and PDTB discourse markers.

Limitations

  • Stated: neither essay writers nor raters were English native speakers, affecting external validity and generalizability; essay topics may not be an unbiased sample; results cannot be certainly generalized beyond ChatGPT-3 and ChatGPT-4 to models like Bard, Alpaca, or Dolly; the models are moving targets; data contamination is possible but argued unproblematic for general topics; subtle hallucinations may have been overlooked; no multiple model configurations were tested due to effort, and the study is restricted to ChatGPT.
  • Stated: human essays came from an online forum and went through at least one revision with native speakers, so they may be above average for non-native speakers; teacher subject combinations were unknown for data protection.
  • Evident from the provided text: Table 4’s header repeats “ChatGPT-3 vs. ChatGPT-4” for two columns, so the exact pairwise p-value mapping is unclear in this rendering.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41598-023-45644-9

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 2 Not valid · 8 Not checked · 83 Own results and interpretations

31 more claims

Create a free account to see all 94 claims and to verify the 4 references still open.

See all claims — free account
  • It may state two separate claims as one. 32
  • Probably not a substantial claim. 34
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

7 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 51 references

Research works printed but not linked: 7

  1. Leahy, S. & Mishra, P. TPACK and the Cambrian explosion of AI. In Society for Information Technology & Teacher Education International Conference, (ed. Langran, E.) 2465–2469 (Association for the Advancement of Computing in Education (AACE), 2023).
  2. Matt Foulis, J. V. & Reed, C. Interactive visualisation of debater identification and characteristics. In Proceedings of the COMMA workshop on Argument Visualisation, COMMA, 1–7 (2020).
  3. Chatzipanagiotidis, S., Giagkou, M. & Meurers, D. Broad linguistic complexity analysis for Greek readability classification. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, 48–58 (Association for Computational Linguistics, Online, 2021).
  4. Stab, C. & Gurevych, I. Annotating argument components and relations in persuasive essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, 1501–1510 (Dublin City University and Association for Computational Linguistics, Dublin, Ireland, 2014).
  5. Siskou, W., Friedrich, L., Eckhard, S., Espinoza, I. & Hautli-Janisz, A. Measuring plain language in public service encounters. In Proceedings of the 2nd Workshop on Computational Linguistics for Political Text Analysis (CPSS-2022) (Potsdam, Germany, 2022).
  6. Holm, S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979).
  7. Freedman, D., Pisani, R. & Purves, R. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York (2007).

Cited by

See all 344

Metadata

Authors
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A.
Affiliations
University of Passau (DE)
Corresponding author
Herbold, Steffen
Venue
Scientific Reports
Published
2023-10-30
Volume
13
Issue
1
Pages
18617-18617
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4388034851
Cited by
338
References
37
Citations by year
2026: 88 · 2025: 135 · 2024: 109 · 2023: 5
FWCI
78.90
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • OPUS (University of Passau) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Universität Passau
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
ChatGPT, generative AI, AI-generated text quality, essay scoring, teacher ratings, linguistic features, writing style, artificial intelligence in education, homework design, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A large-scale comparison of human-written versus ChatGPT-generated essays. https://app.cochat.ai/papers/10.1038/s41598-023-45644-9
The paper
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023). A large-scale comparison of human-written versus ChatGPT-generated essays. Scientific Reports. https://doi.org/10.1038/s41598-023-45644-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.