Dense and Disconnected: Analyzing the Sedimented Style of ChatGPT-Generated Text at Scale
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
ChatGPT and other LLMs are at the forefront of pedagogical considerations in classrooms across the academy. Many studies have spoken to the technology’s capacity to generate one-off texts in a variety of genres. This study complements those by inquiring into its capacity to generate compelling texts at scale. In this study, we quantitatively and qualitatively analyze a small corpus of generated texts in two genres and gauge it against novice and published academic writers along known dimensions of linguistic variation. Theoretically, we position and historicize ChatGPT as a writing technology and consider the ways in which generated text may not be congruent with established trajectories of writing development in higher education. Our study found that generated texts are more informationally dense than authored texts and often read as dialogically closed, “empty,” and “fluffy.” We close with a discussion of potentially explanatory linguistic features, as well as relevant pedagogical implications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“…models released in the late 2010s (e.g., BERT, GPT) moved windows bidirectionally (not only right to left, but also left to right) and implemented changes to improve performance over larger windows (i.e., longer sequences of words) (Devlin et al., 2019; Peters et al., 2018; Radford et al., 2018).”
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 1×
“…models released in the late 2010s (e.g., BERT, GPT) moved windows bidirectionally (not only right to left, but also left to right) and implemented changes to improve performance over larger windows (i.e., longer sequences of words) (Devlin et al., 2019; Peters et al., 2018; Radford et al., 2018).”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“While it is unclear exactly what texts comprise ChatGPT’s training data, Bender et al. (2021) point out that the model is only trained on 7% of English users.”
Variation across Speech and Writing
Biber, D. (1988) · doi:10.1017/cbo9780511621024 · Cited by 5,227
Stance and engagement: a model of interaction in academic discourse
Hyland, K. (2005) · Discourse Studies · doi:10.1177/1461445605050365 · Cited by 1,998
The Unreasonable Effectiveness of Data
Halevy, A., Norvig, P., & Pereira, F. (2009) · IEEE Intelligent Systems · doi:10.1109/mis.2009.36 · Cited by 1,767
Academic Ableism: Disability and Higher Education
Dolmage, J. (2017) · University of Michigan Press eBooks · doi:10.3998/mpub.9708722 · Cited by 884 · cited 1×
“Lastly, as scholars consider the ways inaccessibility is baked into the material and intellectual dimensions of higher education (Dolmage, 2017; Kerschbaum et al., 2017), the nominal style of academic writing has come under needed scrutiny for its capacity to efficiently chunk specific information…”
BIBER, D., GRAY, B., & POONPON, K. (2011) · TESOL Quarterly · doi:10.5054/tq.2011.244483 · Cited by 804
Biber, D., & Conrad, S. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108686136 · Cited by 623
Writing Technology: Studies in the Materiality of Literacy
Haas, C. (1997) · College Composition and Communication · doi:10.2307/358463 · Cited by 392
Research works printed but not linked: 13
- Aull L., 2020, How students write
- Byrd A. (2023). Truth-telling: Critical inquiries on LLMs and the corpus texts that train them. Composition Studies, 51(1), 135–142. https://compositionstudiesjournal.files.wordpress.com/2023/06/byrd.pdf
- 10.1177/0741088317753348 · DOI
- Conrad D., 2001, Variation in English
- Devitt A. J., 2008, Writing genres
- Firth J. R., 1957, Papers in linguistics, 1934-1951
- Giltrow J., 2021, Academic writing
- Halliday M. A. K., 1976, Cohesion in English
- Radford A., 2018, Improving language understanding with unsupervised learning
- 10.58680/ccc19991338 · doi:10.58680/ccc19991338 · Paper page
- S. V. G. (1908). Do your pupils use erasers? Western Teacher: Devoted to Schoolroom Methods. Practical Aids and Usable Materials for Progressive Teachers, 16(5), 175–176. https://books.google.com/books?id=CKOfn_EPJNgC
- Vee A. (2023). Large language models write answers. Composition Studies, 51(1), 176–181. https://compositionstudiesjournal.files.wordpress.com/2023/06/vee.pdf
- Weaver W., 1949, Translation
Cited by
Do LLMs write like humans? Variation in grammatical and rhetorical styles
Reinhart, A., Markey, B., Laudenbach, M., Pantusen, K., Yurko, R., Weinberg, G., & Brown, D. W. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2422455122 · Cited by 107
“These findings corroborate other research that points out the ways these features produce informationally dense prose (28).”
Self-citationDoes ChatGPT Write Like a Student? Engagement Markers in Argumentative Essays
Jiang, F. (., & Hyland, K. (2025) · Written Communication · doi:10.1177/07410883251328311 · Cited by 52
“This “audience blindness” can result in output that, while grammatically conventional and topically relevant, may lack the nuanced involvement features that characterize effective human writing (Markey et al., 2024; Jiang & Hyland, 2025).”
InfluentialMetadiscursive nouns in academic argument: ChatGPT vs student practices
Jiang, F. K., & Hyland, K. (2025) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2025.101514 · Cited by 52
Rhetorical distinctions: Comparing metadiscourse in essays by ChatGPT and students
Jiang, F. (., & Hyland, K. (2025) · English for Specific Purposes · doi:10.1016/j.esp.2025.03.001 · Cited by 44
EAP in a changing world: Towards a new research agenda
Jiang, F. K., & Hyland, K. (2026) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2026.101647 · Cited by 33
Metadata
- Authors
- Markey, B., Brown, D. W., Laudenbach, M., & Kohler, A.
- Affiliations
- Carnegie Mellon University (US)
- New Jersey Institute of Technology (US)
- Corresponding author
- Markey, Ben
- Venue
- Written Communication
- Publisher
- SAGE Publications
- Published
- 2024-08-04
- Volume
- 41
- Issue
- 4
- Pages
- 571-600
- ISSN
- 0741-0883 (ISSN-L)
- 1552-8472
- OpenAlex
- W4401316021
- Cited by
- 27
- References
- 49
- Citations by year
- 2026: 12 · 2025: 15
- FWCI
- 122.74
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Written Communication (publishedVersion)
- Topics
- Social Sciences › Linguistics and Language › Discourse Analysis in Language Studies
- Social Sciences › Education › Artificial Intelligence in Education
- Social Sciences › Education › Linguistics and Education Research
- Keywords
- ChatGPT, large language models, AI-generated text, academic writing, writing development, information density, language variation, novice writers, writing instruction, college writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Dense and Disconnected: Analyzing the Sedimented Style of ChatGPT-Generated Text at Scale. https://app.cochat.ai/papers/10.1177/07410883241263528
- The paper
- Markey, B., Brown, D. W., Laudenbach, M., & Kohler, A. (2024). Dense and Disconnected: Analyzing the Sedimented Style of ChatGPT-Generated Text at Scale. Written Communication. https://doi.org/10.1177/07410883241263528
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.