A large-scale comparison of human-written versus ChatGPT-generated essays
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index; 2 of the 15 sources its claims cite were checked for support.
Abstract
This study compares human-written and ChatGPT-generated argumentative essays and reports differing linguistic characteristics.
Overview
CoChat Summary
TL;DR
In a large-scale comparison, ChatGPT-generated argumentative essays were rated higher than human-written essays across all seven criteria, with ChatGPT-4 best overall (M=4.68 vs ChatGPT-3 4.36 vs humans 3.69 on a 0–6 scale). The models also differed linguistically from students, e.g., more nominalizations and fewer modals/epistemic markers.
Key findings
- ChatGPT-4 had the highest mean rating for every criterion; overall means were 4.68 for ChatGPT-4, 4.36 for ChatGPT-3, and 3.69 for humans. Human vs. ChatGPT-3 effect sizes ranged from 0.52 to 1.15; human vs. ChatGPT-4 from 0.88 to 1.43. ChatGPT-4 vs. ChatGPT-3 differences were significant only for logic and composition, vocabulary and text linking, and complexity, with effect sizes 0.45–0.5.
- Inter-rater agreement was consistently excellent (Cronbach’s α 0.95–0.97) except language mastery, which was good (α=0.89).
- Linguistic markers differed: lexical diversity means were humans 95.72, ChatGPT-3 75.68, ChatGPT-4 108.91; nominalizations 1.06, 1.56, 1.73; modals 10.84, 8.97, 6.12; epistemic markers 0.06, 0.02, 0.00; discourse markers 0.57, 0.52, 0.36. Sentence complexity (depth) means were 5.72, 6.18, 5.94; clause complexity 1.81, 2.31, 2.08.
- Complexity ratings correlated positively with sentence complexity and nominalizations, vocabulary ratings with lexical diversity, and logic ratings negatively with discourse markers; all correlations were significant but weak.
Methods
- Data: 90 argumentative essay topics from Stab and Gurevych’s corpus; human essays came from Essay Forum and were written by non-native high-school students likely in grades 11–13. ChatGPT-3 and ChatGPT-4 browser versions were prompted on 22 March 2023 with “Write an essay with about 200 words on ‘[topic]’” in a zero-shot setting, yielding 270 essays total (90 per source).
- Annotation: 111 teachers completed the questionnaire, producing 658 ratings for 270 essays; each participant was shown six randomly selected essays without source information. Essays were rated on seven criteria from Lower Saxony guidelines using a seven-point Likert scale (0–6), plus self-assessed English skills and rating confidence.
- Analysis: within-subjects design; Cronbach’s α for inter-rater reliability; two-sided Wilcoxon rank-sum tests with Holm-Bonferroni correction; Cohen’s d; Pearson correlations. Computational linguistic measures included MTLD lexical diversity, spaCy dependency depth, clause counts, nominalization suffixes, modals, epistemic markers, and PDTB discourse markers.
Limitations
- Stated: neither essay writers nor raters were English native speakers, affecting external validity and generalizability; essay topics may not be an unbiased sample; results cannot be certainly generalized beyond ChatGPT-3 and ChatGPT-4 to models like Bard, Alpaca, or Dolly; the models are moving targets; data contamination is possible but argued unproblematic for general topics; subtle hallucinations may have been overlooked; no multiple model configurations were tested due to effort, and the study is restricted to ChatGPT.
- Stated: human essays came from an online forum and went through at least one revision with native speakers, so they may be above average for non-native speakers; teacher subject combinations were unknown for data protection.
- Evident from the provided text: Table 4’s header repeats “ChatGPT-3 vs. ChatGPT-4” for two columns, so the exact pairwise p-value mapping is unclear in this rendering.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 2 Not valid · 8 Not checked · 83 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT-4 (ChatGPT web interface with the GPT-4 model) significantly outperforms ChatGPT-3 (ChatGPT web interface with the GPT-3.5 default model) with respect to logical structure, language complexity, vocabulary richness and text linking.”
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Introduction
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Writing styles between humans and generative AI models differ significantly: for instance, the GPT models use more nominalizations and have higher sentence complexity (signaling more complex, ‘scientific’, language), whereas the students make more use of modal and epistemic constructions (which tend to convey speaker attitude).”
- Also stated in: § Abstract · plain
“The writing style of the AI models exhibits linguistic characteristics that are different from those of the human-written essays.”
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Introduction
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the GPT models use more nominalizations and have higher sentence complexity (signaling more complex, ‘scientific’, language), whereas the students make more use of modal and epistemic constructions”
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Introduction
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The linguistic diversity of the NLG models seems to be improving over time: while ChatGPT-3 still has a significantly lower linguistic diversity than humans, ChatGPT-4 has a significantly higher diversity than the students.”
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Introduction
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the GPT models use more nominalizations and have higher sentence complexity (signaling more complex, ‘scientific’, language), whereas the students make more use of modal and epistemic constructions”
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Introduction
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
31 more claims
Create a free account to see all 94 claims and to verify the 4 references still open.
- It may state two separate claims as one. 32
- Probably not a substantial claim. 34
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“In an experiment, English instructors hired to differentiate AI-generated and human-written essays were only able to get 67% accuracy [81].”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [81]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI-generated text tends to outperform student writing on lexical and syntactic sophistication measures while lacking rhetorical hedging, personal voice, and contextual adaptability”
From ARE AI DETECTORS FAIR TO EFL LEARNERS? A LIBRARY RESEARCH ON DETECTION BIAS AND ACADEMIC INTEGRITY (2026) · cites this paper as Herbold et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GenAI can thus support various human tasks such as writing, software development, composing lyrics, and academic research, often at a performance similar to that of humans [6, 11, 26, 28, 50, 72].”
From Generative AI and Creativity: A Systematic Literature Review and Meta-Analysis (2025) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“So far, quantitative comparisons have looked mainly at basic grammar and syntax (5) or features such as word choice, punctuation, sentence length, and so on, finding evidence of some differences between human- and LLM-written text (10–14).”
From Do LLMs write like humans? Variation in grammatical and rhetorical styles (2025) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As LLMs evolve, distinguishing between human and AI-generated text may become more challenging [41].”
From Delving Into PubMed Records: How AI-Influenced Vocabulary has Transformed Medical Writing since ChatGPT (2025) · cites this paper as [41]
Experimental — accuracy not yet measured
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Results
“| Criterion/Linguistic characteristic | Human vs. ChatGPT-3 | ChatGPT-3 vs. ChatGPT-4 | ChatGPT-3 vs. ChatGPT-4 |”
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Materials and methods > Statistical methods
“This results in three comparisons (human vs. ChatGPT-3, human vs. ChatGPT-4, ChatGPT-3 vs. ChatGPT-4)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Materials and methods > Statistical methods
“We use two-sided Wilcoxon-rank-sum tests57 to confirm the significance of the differences between the data sources for each criterion.”
§ A large-scale comparison of human-written versus ChatGPT-generated essays > Results
“Table: Table 4 P-values of the Wilcoxon signed-rank tests adjusted for multiple comparisons using the Holm-Bonferroni method.”
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1×
“While this approach does not directly assess the quality of the output, it serves as a Turing test 31 designed to evaluate whether humans can distinguish between human-and AI-produced output.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Research works printed but not linked: 7
- Leahy, S. & Mishra, P. TPACK and the Cambrian explosion of AI. In Society for Information Technology & Teacher Education International Conference, (ed. Langran, E.) 2465–2469 (Association for the Advancement of Computing in Education (AACE), 2023).
- Matt Foulis, J. V. & Reed, C. Interactive visualisation of debater identification and characteristics. In Proceedings of the COMMA workshop on Argument Visualisation, COMMA, 1–7 (2020).
- Chatzipanagiotidis, S., Giagkou, M. & Meurers, D. Broad linguistic complexity analysis for Greek readability classification. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, 48–58 (Association for Computational Linguistics, Online, 2021).
- Stab, C. & Gurevych, I. Annotating argument components and relations in persuasive essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, 1501–1510 (Dublin City University and Association for Computational Linguistics, Dublin, Ireland, 2014).
- Siskou, W., Friedrich, L., Eckhard, S., Espinoza, I. & Hautli-Janisz, A. Measuring plain language in public service encounters. In Proceedings of the 2nd Workshop on Computational Linguistics for Political Text Analysis (CPSS-2022) (Potsdam, Germany, 2022).
- Holm, S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6, 65–70 (1979).
- Freedman, D., Pisani, R. & Purves, R. Statistics (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York (2007).
Cited by
AI-generated feedback on writing: insights into efficacy and ENL student preference
Escalante, J., Pack, A., & Barrett, A. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00425-2 · Cited by 589
Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025) · Computers & Education · doi:10.1016/j.compedu.2024.105224 · Cited by 531
Opportunities, challenges and school strategies for integrating generative AI in education
Ng, D. T. K., Chan, E. K. C., & Lo, C. K. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100373 · Cited by 326
Bozkurt, A. (2024) · Open Praxis · doi:10.55982/openpraxis.16.1.654 · Cited by 215
AI and Its Impact on Creativity and Diversity: An Empirical Study of LLM-Generated Product Ideas
Terwiesch, C., Meincke, L., Girotra, K., Mollick, E. R., Nave, G., & Ulrich, K. T. (2023) · doi:10.2139/ssrn.4526071 · Cited by 174
Metadata
- Authors
- Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A.
- Affiliations
- University of Passau (DE)
- Corresponding author
- Herbold, Steffen
- Venue
- Scientific Reports
- Published
- 2023-10-30
- Volume
- 13
- Issue
- 1
- Pages
- 18617-18617
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC10616290
- OpenAlex
- W4388034851
- Cited by
- 338
- References
- 37
- Citations by year
- 2026: 88 · 2025: 135 · 2024: 109 · 2023: 5
- FWCI
- 78.90
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- OPUS (University of Passau) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Universität Passau
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- ChatGPT, generative AI, AI-generated text quality, essay scoring, teacher ratings, linguistic features, writing style, artificial intelligence in education, homework design, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A large-scale comparison of human-written versus ChatGPT-generated essays. https://app.cochat.ai/papers/10.1038/s41598-023-45644-9
- The paper
- Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023). A large-scale comparison of human-written versus ChatGPT-generated essays. Scientific Reports. https://doi.org/10.1038/s41598-023-45644-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.