Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 14 of the 17 sources its claims cite were checked for support.

Do LLMs write like humans? Variation in grammatical and rhetorical styles

Open access

Published version of a preprint

Abstract

Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems. As they have advanced, it has become difficult to distinguish their output from human-written text. While past research has found some differences in features such as word choice and punctuation and developed classifiers to detect LLM output, none has studied the rhetorical styles of LLMs. Using several variants of Llama 3 and GPT-4o, we construct two parallel corpora of human- and LLM-written texts from common prompts. Using Douglas Biber’s set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans and between different LLMs. These differences persist when moving from smaller models to larger ones and are larger for instruction-tuned models than base models. This observation of differences demonstrates that despite their advanced abilities, LLMs struggle to match human stylistic variation. Attention to more advanced linguistic features can hence detect patterns in their behavior not previously recognized.

Overview

CoChat Summary

TL;DR

This study compares the writing style of several LLMs to human-authored English texts using Biber’s linguistic features. It finds that instruction-tuned models have a distinct noun-heavy, informationally dense style that does not align with human genre conventions, even when prompted to match informal speech and writing.

Key findings

  • A random forest using Biber features to classify seven text sources in HAP-E achieved 66% test accuracy versus 14% expected by chance; only 4.2% of LLM texts were falsely classified as human, and only 9.8% of human texts were falsely classified as LLMs.
  • Instruction-tuned models preferred present participial clauses, “that” clauses as subjects, nominalizations, and phrasal coordination. GPT-4o used present participial clauses at 5.3 times the human rate (paired Cohen’s d=1.38), “that” clauses as subject 2.6 times as often (d=0.77), nominalizations 2.1 times as often (d=1.23), and phrasal coordination 1.9 times as often (d=0.81); GPT-4o also used agentless passive voice at roughly half the human rate.
  • GPT-4o and GPT-4o Mini used words such as camaraderie, palpable, tapestry, and intricate at more than 100 times the human rate; “tapestry” appeared in 23% of GPT-4o outputs and “amidst” in 27%. Instruction-tuned Llama 3 variants also favored words like camaraderie and palpable, though at lower rates than GPT-4o.
  • Pairwise random forest classifiers distinguishing human text from one specific LLM achieved typical accuracies of 93 to 98%, even when trained on HAP-E and tested on CAP or vice versa; lasso-penalized logistic regression was similar except for Llama base models, which had accuracies around 75%. On M4 arXiv abstracts, classifiers trained on instruction-tuned LLMs classified GPT-3.5 output above chance, but models trained on Llama base outputs attained only 50% accuracy, equal to random guessing.

Methods

  • Two parallel corpora were built from n=12,000 human-authored English texts each: HAP-E from six categories (academic, news, fiction, spoken word, blogs, TV/movie scripts) and CAP from COCA’s eight registers. After removing refusals or unusable answers, there were n=8,290 HAP-E texts and n=9,615 CAP texts with outputs from all LLMs; with two human chunks and six LLM chunks per text, HAP-E comprised n=66,320 chunks and CAP n=76,920 chunks.
  • Six LLMs were used: GPT-4o, GPT-4o Mini, and Meta Llama 3 8B, 70B, 8B Instruct, and 70B Instruct. Each human text was split into two consecutive ~500-word chunks; the first chunk prompted the LLM to write 500 more words in the same style, tone, and diction, and the generated text was compared to the second human chunk.
  • Features came from Douglas Biber’s tagset of 66 linguistic categories. Differences were tested with paired Wilcoxon signed-rank tests with Bonferroni correction; random forest and lasso-penalized logistic regression classifiers were used, with M4 arXiv abstracts generated by GPT-3.5 as a generalizability check.

Limitations

  • Stated: the instruction-tuning processes are not publicly documented, so the precise cause of stylistic differences cannot be determined. The paper also states that generalizability across LLMs or to different registers is difficult, as accuracy dropped significantly on M4 and Llama base classifiers reached only 50% accuracy.
  • Stated: the authors do not have access to untuned versions of GPT, so comparisons between base and instruction-tuned models rely on Llama 3.
  • Evident from the text: the study covers six LLMs, mostly GPT-4o and Llama 3 variants, and only English text; classification accuracy is not perfect, and the authors explicitly say their intention is not to propose another LLM detector.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.2422455122

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1. The LLM text generation workflow. Each human text was split into two chunks of roughly 500 words; the first chunk was used to prompt an LLM to create text that was compared to the second human chunk.
Fig. 1. The LLM text generation workflow. Each human text was split into two chunks of roughly 500 words; the first chunk was used to prompt an LLM to create text that was compared to the second human chunk.CC BY-NC-ND · PubMed Central
Fig. 2. Confusion matrix for a random forest classifying HAP-E texts by their linguistic and rhetorical features, evaluated on the test set (25% of the HAP-E corpus, including n=14,535 human and LLM texts). The block diagonal structure indicates that most classification errors were between different versions of the same LLM, rather than between humans and LLMs.
Fig. 2. Confusion matrix for a random forest classifying HAP-E texts by their linguistic and rhetorical features, evaluated on the test set (25% of the HAP-E corpus, including n=14,535 human and LLM texts). The block diagonal structure indicates that most classification errors were between different versions of the same LLM, rather than between humans and LLMs.CC BY-NC-ND · PubMed Central
Fig. 3. Rate of Biber feature use by different LLMs, relative to the human usage of each feature, for the top 15 most important features in the HAP-E corpus. Note the log scale. GPT-4o and GPT-4o Mini show the largest variation from human texts, while the base variants of Llama 3 most closely resemble human grammar and style. Larger models (Top row) generally show the same stylistic differences as their smaller counterparts (Bottom row), despite performing better on other benchmark tasks. Triangles indicate statistically significant differences from human usage. SI Appendix, Fig. S1 gives the distribution of paired differences.
Fig. 3. Rate of Biber feature use by different LLMs, relative to the human usage of each feature, for the top 15 most important features in the HAP-E corpus. Note the log scale. GPT-4o and GPT-4o Mini show the largest variation from human texts, while the base variants of Llama 3 most closely resemble human grammar and style. Larger models (Top row) generally show the same stylistic differences as their smaller counterparts (Bottom row), despite performing better on other benchmark tasks. Triangles indicate statistically significant differences from human usage. SI Appendix, Fig. S1 gives the distribution of paired differences.CC BY-NC-ND · PubMed Central
Fig. 4. Rates of word use by different LLMs (per 1,000 words) compared to the human use of each word in chunk 2, in the HAP-E corpus (log scale). Includes all words used more than once per million words in chunk 2. Words are lemmatized to group together inflected forms. Words on the diagonal are used equally often in human and LLM texts. Dashed blue lines indicate the range between 10× more and 10× less than human use. Note that the instruction-tuned models show more variation from the diagonal, indicating more deviation in vocabulary use relative to humans.
Fig. 4. Rates of word use by different LLMs (per 1,000 words) compared to the human use of each word in chunk 2, in the HAP-E corpus (log scale). Includes all words used more than once per million words in chunk 2. Words are lemmatized to group together inflected forms. Words on the diagonal are used equally often in human and LLM texts. Dashed blue lines indicate the range between 10× more and 10× less than human use. Note that the instruction-tuned models show more variation from the diagonal, indicating more deviation in vocabulary use relative to humans.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

4 Valid · 0 Not valid · 1 Not checked · 51 Own results and interpretations

40 more claims

Create a free account to see all 56 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 3
  • It may state two separate claims as one. 9

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 30 references

Cited by

See all 107

Metadata

Authors
Reinhart, A., Markey, B., Laudenbach, M., Pantusen, K., Yurko, R., Weinberg, G., & Brown, D. W.
Affiliations
  • Carnegie Mellon University (US)
  • New Jersey Institute of Technology (US)
Corresponding author
Reinhart, Alex
Venue
Proceedings of the National Academy of Sciences
Published
2025-02-18
Volume
122
Issue
8
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4407695356
Cited by
107
References
30
Citations by year
2026: 82 · 2025: 25
FWCI
27.29
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • Carnegie Mellon University
  • College of Humanities and Social Sciences, United Arab Emirates University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, LLM-generated text, human-written text, rhetorical style, grammatical style, stylistic variation, Llama 3, GPT-4o, parallel corpus, model size, instruction tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Do LLMs write like humans? Variation in grammatical and rhetorical styles. https://app.cochat.ai/papers/10.1073/pnas.2422455122
The paper
Reinhart, A., Markey, B., Laudenbach, M., Pantusen, K., Yurko, R., Weinberg, G., & Brown, D. W. (2025). Do LLMs write like humans? Variation in grammatical and rhetorical styles. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2422455122

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.