Contrasting Linguistic Patterns in Human and LLM-Generated News Text

May combine two claims.

LLM outputs use more numbers, symbols and auxiliaries than human texts.

“LLM outputs use more numbers, symbols and auxiliaries (suggesting objective language) than human texts, as well as more pronouns.”

May combine two claims.

The tested LLMs generate sentences of 10-30 tokens more frequently than humans, while humans produce longer sentences more frequently.

“Specifically, the models exhibit a higher frequency of sentence generation within the 10 to 30 token range compared to humans, whereas humans tend to produce longer sentences with greater frequency.”

May combine two claims.

Across LLaMa models, fear increases as the number of parameters grows (from LLaMa 13B), making them more akin to human texts.

“Across LLaMa models, fear increases as the number of parameters grows (from LLaMa 13B), making them more akin to human texts.”

May combine two claims.

LLMs have a tendency to use more auxiliary verbs and verb phrases and less noun and prepositional phrases than humans.

“while they have a tendency to use more auxiliary verbs and verb phrases and less noun and prepositional phrases.”

May combine two claims.

Human texts have higher optimality scores than LLM texts, indicating greater dependency length minimization.

“Results indicate that the distribution of values is almost identical between all of the LLMs, but human texts show noticeably larger values. This means human texts are more optimized in terms of dependency lengths, i.e. they have shorter dependencies than expected by a larger margin than those generated by the LLMs.”

May combine two claims.

A substantial majority of lead paragraphs are classified as neutral, accounting for over 50% of texts across all models and human samples, with LLM text slightly more neutral.

“As anticipated in journalistic texts, a substantial majority of the lead paragraphs are classified as neutral. This category accounts for over 50% of the texts across all models and human-generated samples, with the LLM-generated text demonstrating a slightly higher inclination towards neutrality.”

May combine two claims.

There is minimal divergence in adverb phrase (ADVP) frequency except for Falcon 7B, which shows a great difference; LLMs show a more pronounced propensity for verb phrases (VP).

“there is minimal divergence in the frequency of adverb phrases (ADVP) except for Falcon 7B, which shows a great difference with human and LLM-generated texts, the latter exhibiting a more pronounced propensity for verb phrases (VP).”

May combine two claims.

LLM-generated sentences have less variation and more uniform lengths than human-generated sentences, which display a wider range of lengths and greater diversity.

“The six LLMs exhibit a similar distribution across different sentence lengths, presenting less variation when compared to human-generated sentences, which display a wider range of lengths and greater diversity.”

May combine two claims.

At the morphosyntactic level, discernible distinctions were observed between human and machine-generated texts, with the latter preferring parts of speech displaying objectivity (such as symbols or numbers) and using substantially fewer adjectives.

“at the morphosyntactic level, discernible distinctions were observed between human and machine-generated texts, the latter having a preference for parts of speech displaying (a sense of) objectivity - such as symbols or numbers - while using substantially fewer adjectives.”

May combine two claims.

Despite similar VERB UPOS tag frequency, LLMs exhibit a more pronounced propensity for verb phrases (VP), consistent with increased auxiliary verbs.

“Despite the similar frequency of the VERB UPOS tag in human and LLM-generated texts, the latter exhibit a more pronounced propensity for verb phrases (VP), consistent with the increased use of auxiliary verbs”

May combine two claims.

Mistral and LLaMa show similar use of grammatical categories, with Mistral being the model that resembles humans the most.

“Comparing LLM families, Mistral and LLaMa show a similar use of grammatical categories, with Mistral being the model that resembles humans the most.”

May combine two claims.

For LLaMa models, larger model size is associated with greater similarity to humans in UPOS usage, but differences between model sizes are smaller than differences between models and humans.

“Regarding model size, the larger the model, the greater the similarity with humans. Nevertheless, differences between differently-sized models are much smaller than between models and humans.”

May combine two claims.

Dependency lengths and their distributions are nearly identical for all LLMs except Falcon, which uses longer dependencies than the rest and resembles human texts more in this respect.

“Results indicate that dependency lengths and their distributions are nearly identical for all the LLMs except Falcon, which uses longer dependencies than the rest of the models and resembles more the human texts in this respect.”

May combine two claims.

Human-generated sentences are longer on average than LLM-generated sentences, but average sentence constituent length is greater for LLMs than for humans.

“While human-generated sentences, on average, surpass the length of those generated by LLMs, the average length of a sentence constituent for LLMs is observed to be greater than for humans.”

May combine two claims.

Mistral comes closest to expressing emotions in a way similar to humans, and Falcon expresses more joy and less anger and surprise than the rest of the models.

“When looking at families, Mistral comes closest to expressing emotions in a way similar to humans, and Falcon expresses more joy and less anger and surprise than the rest of the models.”

May combine two claims.

Regarding model families, results for constituent types are similar to dependencies and POS tags, but model size trends are less obvious.

“Regarding model families, results are similar to those of dependencies and POS tags, but when looking at model size, previous trends are less obvious.”

The quote only partly states this claim.

LLMs use symbols and numbers more than humans.

“the language models exhibit a pronounced inclination towards relying on categories such as symbols or numbers”

May combine two claims.

LLMs produce many sad texts, a passive but negative emotion, yet less toxic than anger or fear.

“The LLMs also produce many sad texts, a passive but negative emotion, yet less toxic6 than emotions such as anger or fear.”

May combine two claims.

LLM texts have higher ratios of aux, copula, and nominal subject (nsubj) dependencies than human texts.

“we observed higher ratios for other dependency types, such as aux (for which the use of auxiliary verbs was also significantly higher according to the UPOS analysis), copula and nominal subjects (nsubj).”

May combine two claims.

LLM-generated texts are consistent among themselves in UPOS tag usage but differ from human texts.

“Overall, the behavior of LLMs and their generated text tends to be consistent among themselves, yet shows differences when compared to human behavior, i.e., they exhibit in some cases a greater or lesser use of certain grammatical categories.”

May combine two claims.

Among the tested LLMs, LLaMa models have lexical diversity closest to human texts, followed by Mistral, with Falcon having the lowest lexical diversity.

“The model family that comes closer to human texts is LLaMa, obtaining similar scores for every model size. Then Mistral comes close, and Falcon is last, exhibiting the lowest lexical diversity by far according to STTR.”

• Also stated in: § Morphosyntactic analysis · plain

  “human texts exhibit the highest lexical diversity, closely followed by LLaMa models, then Mistral and, far away, Falcon.”

May combine two claims.

Falcon 7B, although still biased toward male pronouns, reduces the male pronoun bias by 7.5% relative to human texts.

“Falcon 7B, which, although still heavily biased towards male pronouns, reduces the bias by 7.5%.”

The quote only partly states this claim.

Humans use nouns and adjectives more than LLMs.

“humans exhibit a preference for using certain kinds of content words, such as nouns and adjectives.”

May combine two claims.

LLM texts exhibit similar uses of syntactic dependencies among themselves, with Falcon being the most distinct, while all differ from human-written news.

“We observe similar trends to the previous sections, with LLM texts exhibiting similar uses of syntactic dependencies among themselves, with Falcon being the most distinct model, while all of them present differences compared to human-written news.”

Classified as probably not a substantial claim.

The greater use of numbers, symbols and auxiliaries by LLMs suggests objective language.

“LLM outputs use more numbers, symbols and auxiliaries (suggesting objective language) than human texts, as well as more pronouns.”

Classified as probably not a substantial claim.

The study's focus on English implies that results should be interpreted with caution when applied outside the English-language news domain.

“While this choice is common in various language-related fields, including cognitive science (Blasi et al. 2022), it implies interpreting our results with caution when applying them outside the context of the English language and the news domain.”

• cites Blasi et al. 2022 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The LLMs used are English-centric: LLaMa's dataset comprises over 70% English content and Falcon's over 80%.

“Firstly, the LLMs we use (as detailed in Sect. 3.2) are English-centric. LLaMa’s dataset comprises over 70% English content, and Falcon’s even higher at over 80%.”

• cites Touvron et al. 2023Not checked yet

  Not checked in the free run.

• cites Almazrouei et al. 2023Not checked yet

  Not checked in the free run.

May combine two claims.

The evaluation dataset was generated using news published after the release date of the models used, ensuring the models had no exposure to the news headlines and content during pre-training.

“We generate the evaluation dataset relying on news published after the release date of the models that we will use in this work. This strategy ensures that they did not have exposure to the news headlines and their content during pre-training.”

• AssumptionRests on an assumption: News published after a model's release date could not have been in its pre-training data.

May combine two claims.

Stanza's performance for segmentation, tokenization, PoS tagging, and dependency and constituent parsing on English news text is high enough for applications and can be superior to human annotations.

“For these tasks, and in particular for the case of English and news text, the performance is high enough to be used for applications (Manning 2011; Berzak et al. 2016), and it can be even superior to that obtained by human annotations.”

• cites Manning 2011Not checked yet

  Not checked in the free run.

• cites Berzak et al. 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLaMa exhibits superior performance in zero and few-shot scenarios compared to larger models like GPT-3, PaLM, and Chinchilla.

“In comparison to larger models like GPT-3 (Brown et al. 2020), PaLM (Chowdhery et al. 2023), and Chinchilla (Hoffmann et al. 2022), LLaMa exhibits superior performance in zero and few-shot scenarios.”

• cites Brown et al. 2020Not checked yet

  Not checked in the free run.

• cites Chowdhery et al. 2023Not checked yet

  Not checked in the free run.

• cites Hoffmann et al. 2022Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The stochastic nature of LLMs could streamline outputs, reducing extreme events and resulting in less variation and more uniform sentence lengths.

“These results might be explained by considering the stochastic nature of the models, which could streamline outputs, thereby reducing the occurrence of extreme events. This may result in less variation and more uniform sentence lengths in LLM-generated text”

Classified as probably not a substantial claim.

To ensure the novelty of texts and address memorization concerns, the study fed the LLMs headlines from news articles published after the release date of the models.

“To ensure the novelty of texts and address memorization concerns, we fed the LLMs headlines from news articles published after the release date of the models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Mistral v0.1 surpasses larger LLaMa models in various benchmarks despite its smaller size.

“Mistral v0.1 surpasses larger LLaMa models in various benchmarks despite its smaller size.”

• cites Jiang et al. 2023Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The cause of the gender disparity could be related to LLMs writing more generic pronouns than humans, which would exacerbate the disparity between male and female references as presented in the news.

“This analysis is limited by the fact that the cause of this disparity could be related to the LLMs writing more generic pronouns than the humans, hence exacerbating the disparity between male and female references as presented in the news.”

• AssumptionRests on an assumption: In English, masculine pronouns are more widely used as generic pronouns than female or neutral pronouns.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

The main source contributing to increased fear in larger LLaMa models might be pre-processing steps such as removing toxic content from data.

“we hypothesize the main source contributing to this issue might be some pre-processing steps used for the LLaMa models, such as removing toxic content from its data.”

• AssumptionRests on an assumption: LLaMa version 1 models were not fine-tuned with reinforcement learning with human feedback.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The observed difference in emotions between human and LLM-generated texts may be related to the same regularization effect that affects UPOS and sentence length distribution.

“This difference in emotions may be related to the same regularization effect that affects UPOS and sentence length distribution.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The probabilistic nature of LLMs can blur distinctions between writing styles and genres, resulting in register leveling.

“The probabilistic nature of LLMs can blur the distinctions between writing styles and genres, resulting in what is known as register leveling.”

• AssumptionRests on an assumption: Humans typically use a specific writing style based on the genre.

Classified as probably not a substantial claim.

Human language dependencies tend to be much shorter than expected by chance (dependency length minimization).

“It has been observed in human language that dependencies tend to be much shorter than expected by chance, a phenomenon known as dependency length minimization (Ferrer-i-Cancho 2004; Futrell et al. 2015).”

• cites Ferrer-i-Cancho 2004 sourceNot checked yet

  Not checked in the free run.

• cites Futrell et al. 2015 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The study compared linguistic patterns in texts produced by humans and machines under controlled conditions.

“This paper presented a comprehensive study on linguistic patterns in texts produced by both humans and machines, comparing them under controlled conditions.”

May combine two claims.

LLaMa's technical report mentioned an increase in model toxicity as they scaled up in size despite using the same pre-processing, which is coherent with our findings.

“Yet, LLaMa’s technical report (Touvron et al. 2023) mentioned an increase in model toxicity as they scaled up in size despite using the same pre-processing in all cases, which is coherent with our findings.”

• cites Touvron et al. 2023Not checked yet

  Not checked in the free run.

May combine two claims.

The dependency length finding holds across every sentence length bin for Falcon and all but the first bin for human texts, making it orthogonal to sentence length variation.

“This finding holds true for every sentence length bin for Falcon, and for all but the first (length 1–10) in the case of human texts, so we can be reasonably sure that it is orthogonal to the variation in sentence length distribution between human and LLM texts described earlier.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The LLaMa models are trained on a diverse mix of data sources and domains, predominantly in English.

“The LLaMa models are trained on a diverse mix of data sources and domains, predominantly in English, as detailed in Table 1.”

• cites Touvron et al. 2023Not checked yet

  Not checked in the free run.