The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 14 of the 17 sources its claims cite were checked for support.
Do LLMs write like humans? Variation in grammatical and rhetorical styles
Open accessPublished version of a preprint
Abstract
Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems. As they have advanced, it has become difficult to distinguish their output from human-written text. While past research has found some differences in features such as word choice and punctuation and developed classifiers to detect LLM output, none has studied the rhetorical styles of LLMs. Using several variants of Llama 3 and GPT-4o, we construct two parallel corpora of human- and LLM-written texts from common prompts. Using Douglas Biber’s set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans and between different LLMs. These differences persist when moving from smaller models to larger ones and are larger for instruction-tuned models than base models. This observation of differences demonstrates that despite their advanced abilities, LLMs struggle to match human stylistic variation. Attention to more advanced linguistic features can hence detect patterns in their behavior not previously recognized.
Overview
CoChat Summary
TL;DR
This study compares the writing style of several LLMs to human-authored English texts using Biber’s linguistic features. It finds that instruction-tuned models have a distinct noun-heavy, informationally dense style that does not align with human genre conventions, even when prompted to match informal speech and writing.
Key findings
- A random forest using Biber features to classify seven text sources in HAP-E achieved 66% test accuracy versus 14% expected by chance; only 4.2% of LLM texts were falsely classified as human, and only 9.8% of human texts were falsely classified as LLMs.
- Instruction-tuned models preferred present participial clauses, “that” clauses as subjects, nominalizations, and phrasal coordination. GPT-4o used present participial clauses at 5.3 times the human rate (paired Cohen’s d=1.38), “that” clauses as subject 2.6 times as often (d=0.77), nominalizations 2.1 times as often (d=1.23), and phrasal coordination 1.9 times as often (d=0.81); GPT-4o also used agentless passive voice at roughly half the human rate.
- GPT-4o and GPT-4o Mini used words such as camaraderie, palpable, tapestry, and intricate at more than 100 times the human rate; “tapestry” appeared in 23% of GPT-4o outputs and “amidst” in 27%. Instruction-tuned Llama 3 variants also favored words like camaraderie and palpable, though at lower rates than GPT-4o.
- Pairwise random forest classifiers distinguishing human text from one specific LLM achieved typical accuracies of 93 to 98%, even when trained on HAP-E and tested on CAP or vice versa; lasso-penalized logistic regression was similar except for Llama base models, which had accuracies around 75%. On M4 arXiv abstracts, classifiers trained on instruction-tuned LLMs classified GPT-3.5 output above chance, but models trained on Llama base outputs attained only 50% accuracy, equal to random guessing.
Methods
- Two parallel corpora were built from n=12,000 human-authored English texts each: HAP-E from six categories (academic, news, fiction, spoken word, blogs, TV/movie scripts) and CAP from COCA’s eight registers. After removing refusals or unusable answers, there were n=8,290 HAP-E texts and n=9,615 CAP texts with outputs from all LLMs; with two human chunks and six LLM chunks per text, HAP-E comprised n=66,320 chunks and CAP n=76,920 chunks.
- Six LLMs were used: GPT-4o, GPT-4o Mini, and Meta Llama 3 8B, 70B, 8B Instruct, and 70B Instruct. Each human text was split into two consecutive ~500-word chunks; the first chunk prompted the LLM to write 500 more words in the same style, tone, and diction, and the generated text was compared to the second human chunk.
- Features came from Douglas Biber’s tagset of 66 linguistic categories. Differences were tested with paired Wilcoxon signed-rank tests with Bonferroni correction; random forest and lasso-penalized logistic regression classifiers were used, with M4 arXiv abstracts generated by GPT-3.5 as a generalizability check.
Limitations
- Stated: the instruction-tuning processes are not publicly documented, so the precise cause of stylistic differences cannot be determined. The paper also states that generalizability across LLMs or to different registers is difficult, as accuracy dropped significantly on M4 and Llama base classifiers reached only 50% accuracy.
- Stated: the authors do not have access to untuned versions of GPT, so comparisons between base and instruction-tuned models rely on Llama 3.
- Evident from the text: the study covers six LLMs, mostly GPT-4o and Llama 3 variants, and only English text; classification accuracy is not perfect, and the authors explicitly say their intention is not to propose another LLM detector.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
4 Valid · 0 Not valid · 1 Not checked · 51 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similar to past research (14), we find that LLMs also favor specific vocabulary.”
§ Do LLMs write like humans? Variation in grammatical and rhetorical styles > Results > Differences in Style and Vocabulary.
What the sources say
- Supportedcites [14] W. Liang et al., “Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews” in Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, R. Salakhutdinov et al., Eds. (PMLR, 2024), vol. 235, pp. 29575–29620. source
The Introduction reports validation experiments showing that specific adjectives, including “commendable,” occur disproportionately more often in AI-generated text than in human-written reviews. This supports the claim that LLMs favor specific vocabulary in the peer-review setting studied. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Though not definitive, these results suggest there are indeed structural differences between human- and LLM-written text.”
§ Do LLMs write like humans? Variation in grammatical and rhetorical styles > Abstract
What the sources say
- Supportedcites [10] R. Tang, Y. N. Chuang, X. Hu, The science of detecting LLM-generated text. Commun. ACM 67, 50– 59 (2024). source
The “Linguistic patterns” section describes differences in style, coherence, and repetition between human- and LLM-written text. The “Statistical disparities” section also describes differences in word-ranking patterns and perplexity. The paper notes limits to these signals, consistent with the citing sentence’s “not definitive” qualification. (LLM, conf=0.92)
- Supportedcites [11] L. Fröhling, A. Zubiaga, Feature-based detection of automated language models: Tackling GPT-2, GPT-3 and Grover. PeerJ Comput. Sci. 7, e443 (2021). PMC8049133 33954234 source
The Methodology > Features section describes expected differences in sentence structure and word choice between human and language-model text. The Results > Feature-set classifiers section reports that syntactic features distinguish the two across the tested datasets, supporting the citing sentence’s tentative conclusion. (LLM, conf=0.91)
- Supportedcites [12] A. Muñoz-Ortiz, C. Gómez-Rodríguez, D. Vilares, Contrasting linguistic patterns in human and LLM-generated news text. Artif. Intell. Rev. 57, 265 (2024). 39328400 PMC11422446 source
The Conclusion reports discernible morphosyntactic distinctions and variations in syntactic structures between human-written and LLM-generated texts. This supports the citing sentence’s hedged claim; the paper’s comparison concerns English news text and the models it tested. (LLM, conf=0.93)
- Supportedcites [13] S. Herbold, A. Hautli-Janisz, U. Heuer, Z. Kikteva, A. Trautsch, A large-scale comparison of human-written versus ChatGPT-generated essays. Sci. Rep. 13, 18617 (2023). 37903836 PMC10616290 source
The Discussion says ChatGPT-generated essays are highly structured and follow a more rigid pattern, while human-written essays are looser. This supports the citing sentence’s hedged claim of structural differences between human- and LLM-written text. (LLM, conf=0.92)
- Supportedcites [14] W. Liang et al., “Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews” in Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, R. Salakhutdinov et al., Eds. (PMLR, 2024), vol. 235, pp. 29575–29620. source
The Introduction says LLM-generated text can be indistinguishable from human writing in individual cases but shows contrasting trends at the corpus level. The Homogenization Effect section reports that reviews with more estimated LLM use tended to be more similar to other reviews, supporting the hedged claim about differences in text patterns. (LLM, conf=0.88)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, GPT-4o uses the agentless passive voice at roughly half the rate as human texts—but in each case, the Llama base models use these features at rates more closely matching humans.”
§ Do LLMs write like humans? Variation in grammatical and rhetorical styles > Abstract
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A third major finding of this work is significant success of Biber’s tagset in modeling and classifying text.”
§ Do LLMs write like humans? Variation in grammatical and rhetorical styles > Discussion
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A random forest classifier using the Biber features to distinguish between the seven text sources in HAP-E (human chunk 2 and the six LLMs) achieved a test accuracy of 66%, compared to an expected accuracy of 14% from random guessing.”
§ Do LLMs write like humans? Variation in grammatical and rhetorical styles > Results > Classifying Text by Source.
plainn = 14,535- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
40 more claims
Create a free account to see all 56 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 3
- It may state two separate claims as one. 9
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Do LLMs write like humans? Variation in grammatical and rhetorical styles > Results > Differences in Style and Vocabulary.
“GPT-4o and 4o Mini use words like camaraderie, palpable, tapestry, and intricate at more than 100 times the rate of humans”
§ Do LLMs write like humans? Variation in grammatical and rhetorical styles > Results > Differences in Style and Vocabulary.
“| palpable | 95 | ignite | 122 | camaraderie | 24 | waft | 28 | denis | 14 | worcester | 11 |”
Citations
Cites
Variation across Speech and Writing
Biber, D. (1988) · doi:10.1017/cbo9780511621024 · Cited by 5,227
Miller, C. R. (1984) · Quarterly Journal of Speech · doi:10.1080/00335638409383686 · Cited by 2,907
Biber, D., & Conrad, S. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511814358 · Cited by 1,980
Dimensions of Register Variation: A Cross-Linguistic Comparison
Biber, D. (1995) · Cambridge University Press eBooks · doi:10.1017/cbo9780511519871 · Cited by 1,270
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
A large-scale comparison of human-written versus ChatGPT-generated essays
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-45644-9 · Cited by 338
M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection
Wang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A., Whitehouse, C., Afzal, O. M., Mahmoud, T. M., Sasaki, T., et al. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-long.83 · Cited by 235
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 206
Barman, D., Guo, Z., & Conlan, O. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100545 · Cited by 189 · cited 1×
“Each human text was split into two chunks of roughly 500 words; the first chunk was used to prompt an LLM to create text that was compared to the second human chunk.”
The emergence of economic rationality of GPT
Chen, Y., Liu, T. X., Shan, Y., & Zhong, S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2316205120 · Cited by 164
Cited by
Mansoor, I., Abdullah, M., Rizwan, M. D., & Fraz, M. M. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00403-0 · Cited by 55
“Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems[15].”
Zhang, M. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1641212 · Cited by 21
Survey-taking AI tools surpass human abilities. Here’s what we can do about it
Panizza, F., Kyrychenko, Y., & Roozenbeek, J. (2026) · Nature · doi:10.1038/d41586-026-00386-2 · Cited by 12
Idan, D., Shitrit, I. B., Volevich, M., Binyamin, Y., Nassar, R., Nassar, M., Abelson, N., Zlotnik, A. A., & Einav, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-21689-w · Cited by 16
“As LLM capabilities continue to develop, future studies should explicitly evaluate how features such as RAG, advanced reasoning 38 and differences in writing style and flow 39,40,41 influence output accuracy, physician workflow, and overall trust in these tools.”
A Comparison of DeepSeek and other LLMs
Gao, T., Jin, J., Ke, Z. T., & Moryoussef, G. (2026) · The American Statistician · doi:10.1080/00031305.2025.2611010 · Cited by 70
“Existing studies have shown that AI-generated text and human-written text have different characteristics (e.g., Reinhart et al. 2025).”
Metadata
- Authors
- Reinhart, A., Markey, B., Laudenbach, M., Pantusen, K., Yurko, R., Weinberg, G., & Brown, D. W.
- Affiliations
- Carnegie Mellon University (US)
- New Jersey Institute of Technology (US)
- Corresponding author
- Reinhart, Alex
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2025-02-18
- Volume
- 122
- Issue
- 8
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC11874169
- OpenAlex
- W4407695356
- Cited by
- 107
- References
- 30
- Citations by year
- 2026: 82 · 2025: 25
- FWCI
- 27.29
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Carnegie Mellon University
- College of Humanities and Social Sciences, United Arab Emirates University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, LLM-generated text, human-written text, rhetorical style, grammatical style, stylistic variation, Llama 3, GPT-4o, parallel corpus, model size, instruction tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Do LLMs write like humans? Variation in grammatical and rhetorical styles. https://app.cochat.ai/papers/10.1073/pnas.2422455122
- The paper
- Reinhart, A., Markey, B., Laudenbach, M., Pantusen, K., Yurko, R., Weinberg, G., & Brown, D. W. (2025). Do LLMs write like humans? Variation in grammatical and rhetorical styles. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2422455122
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.