The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index; none of the 1 sources its claims cite is checked for support yet.
Exposing Pink Slime Journalism: Linguistic Signatures and Robust Detection Against LLM-Generated Threats
Open accessAbstract
The local news landscape, a vital source of reliable information for 28 million Americans, faces a growing threat from Pink Slime Journalism, a low-quality, auto-generated articles that mimic legitimate local reporting.Detecting these deceptive articles requires a fine-grained analysis of their linguistic, stylistic, and lexical characteristics.In this work, we conduct a comprehensive study to uncover the distinguishing patterns of Pink Slime content and propose detection strategies based on these insights.Beyond traditional generation methods, we highlight a new adversarial vector: modifications through large language models (LLMs).Our findings reveal that even consumer-accessible LLMs can significantly undermine existing detection systems, reducing their performance by up to 40% in F1-score.To counter this threat, we introduce a robust learning framework specifically designed to resist LLM-based adversarial attacks and adapt to the evolving landscape of automated pink slime journalism, and showed and improvement by up to 27%.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper analyzes linguistic and stylistic signatures that distinguish Pink Slime (PS) journalism from legitimate local news and shows that consumer-accessible LLMs can substantially degrade existing PS detectors. It proposes a continual-learning adaptation framework that improves robustness to LLM-modified PS articles while limiting forgetting on original data.
Key findings
- PS articles are shorter and simpler than legitimate local news: 8.8 vs. 22.97 sentences per article, 60.10% vs. 44.34% simple sentences, 75 vs. 65 adjectives per 1,000 words, and lower lexical richness (RTTR 7.26 vs. 10.16 for local news and 10.21 for national news).
- Transformer fine-tuning performed best for PS detection: BERT reached 89.31% accuracy and 89.05% F1 in Table 2, described as slightly better than XLNet and Flan-T5.
- LLM paraphrasing attacks reduced detection performance: Qwen caused a 28.34% F1 drop, while GPT-4o-mini and Claude-3.5-Haiku reduced F1 by an average of 40% (Table 3: 48.15 and 49.55 F1, respectively, against a baseline of 89%).
- Continual learning improved adversarial detection: without LLM-modified training, F1 on the adversarial test set dropped to 49%; adding 50% LLM-modified samples boosted F1 by 22% uncontrolled and 20% controlled, and 100% reached 75% uncontrolled and 74% controlled, with only 3% and 1% original-distribution F1 decline, respectively.
- PS articles are not consistently classified as fake news: for 60% of PS articles, none of three SOTA fake-news models classified them as fake, and only 7% were unanimously classified as fake; accuracy against local news was 53%, 51%, and 56% for BERT, Flan-T5, and Stella en v5.
Methods
- Used the NELA-PS dataset of 7.9 million articles from 1,093 outlets (2021–2023), randomly sampled 40,000 articles, deduplicated with all-MiniLM-L6-v2 embeddings at cosine similarity 0.8, reducing size by 76.25% and retaining 9,473 PS articles; processed local news from Horne et al. (2022) to 10,000 legitimate samples with 9.81% reduction; also included LIAR and NELA-GT-2021.
- Compared handcrafted features (sentence/word counts, RTTR/CTTR/MTLD, simple sentences, POS and dependency co-occurrence, dependency parsing, readability, POS counts) using XGBoost, Random Forest, and SVM; fine-tuned BERT, XLNet, and Flan-T5; and tested embedding plus fully connected models. Train-test splits used DBSCAN on t-SNE representations for PS and random 80-20 splits for local news, repeated three times.
- Simulated adversarial PS by LLM paraphrasing with open-source and closed-source models, then applied continual learning by incrementally fine-tuning BERT at 1/100th learning rate on mixtures of original local news, LLM-modified PS, and original PS replay (50% of the PS sample), comparing controlled and uncontrolled settings.
Limitations
- The paper states that it does not conduct fact-checking and relies on provided labels as-is, so it cannot make claims about the factual accuracy of the analyzed PS articles; it also notes that although NELA-PS contains over 8 million entries, the analysis was limited to a 40,000-article subset.
- Evident from the text: adversarial evaluation is limited to a specific set of LLMs and paraphrasing prompts, and detection results are based on sampled/deduplicated data, so robustness may not generalize to all LLM-generated modifications or the full PS ecosystem.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 1 cited sources checked · 0 supported · 0 flagged
- Findingown data
PS articles are significantly shorter than legitimate LN articles, averaging 8.8 vs. 22.97 sentences per article (p = 0.00).
“PS articles are significantly shorter than legitimate LN articles, averaging 8.8 vs. 22.97 sentences per article (p = 0.00; Fig. 2a).”
Quote in the paperSubstantial claim - Findingown data
For 60% of Pink Slime articles, none of the three fake news detection models classifies them as fake news.
“Our findings reveal that for 60% of Pink Slime articles, none of the models classifies them as fake news (Figure 7).”
Quote in the paperSubstantial claim - Findingown data
The proposed continual learning framework improves detection under adversarial conditions by up to 27%.
“This approach mitigates catastrophic forgetting and improves detection under adversarial conditions by up to 27%.”
Quote in the paperSubstantial claim - Findingown data
In the absence of LLM-modified PS samples during training, F1-score on the adversarial test set drops substantially to 49%.
“In the absence of any LLM-modified PS samples during training, the F1-score on the adversarial test set drops substantially to 49%”
Quote in the paperSubstantial claim - Findingown data
Even consumer-accessible LLMs can significantly undermine existing detection systems, reducing performance by up to 40% in F1-score.
“Our findings reveal that even consumer-accessible LLMs can significantly undermine existing detection systems, reducing their performance by up to 40% in F1-score.”
Quote in the paperSubstantial claim
- Findingown data
The dense micro-clusters correspond to closely related articles covering topics such as stock performance, crime reports, and donation news.
“Upon further investigation, these clusters correspond to closely related articles covering topics such as stock performance, crime reports, donation news, and similar themes.”
Quote in the paperSubstantial claim - Findingown data
In only 7% of cases do all three models unanimously identify Pink Slime articles as fake news.
“Conversely, in only 7% of cases does all three models unanimously identify Pink Slime articles as fake news.”
Quote in the paperSubstantial claim - Findingown data
Closed-source models GPT-4o-mini and Claude-3.5-Haiku reduced F1-score by an average of 40%.
“Closed-source models were significantly more potent, with both GPT-4o-mini and Claude-3.5-Haikureducing the F1-score by an average of 40%”
Quote in the paperSubstantial claim - Findingown data
PS articles exhibit a distinct pattern of dense micro-clusters scattered across the 2D embedding representation.
“As shown in Figure 3, PS articles exhibit a distinct pattern of dense micro-clusters scattered across the 2D representation.”
Quote in the paperSubstantial claim - Findingown data
PS articles show lower lexical richness than LN and national news, with RTTR 7.26 vs. 10.16 and 10.21.
“We used the Root-Type-Token Ratio (RTTR) (Guiraud, 1954), and found PS articles contain much less richness (RTTR = 7.26) than LN (10.16) and national news (10.21)”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Guiraud, 1954
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Both models demonstrate agreement in assigning high importance to features such as unique noun phrases and sentence count.
“We found that both models demonstrate agreement in assigning high importance to features such as”
Quote in the paperNot a substantial claim - Findingown data
PS articles use more simple sentences than LN (60.10% vs. 44.34%, p = 0.00).
“PS also uses more simple sentences (60.10% vs. 44.34%,p = 0.00; Fig. 2b)”
Quote in the paperSubstantial claim - Findingown data
When comparing Pink Slime articles to local news, classification accuracy across models was relatively low, achieving 53%, 51%, and 56% for BERT, Flan-T5, and Stella en v5.
“When comparing Pink Slime articles to local news, the classification accuracy across the models was relatively low, achieving 53%, 51%, and 56% for BERT, Flan-T5, and Stella en v5, respectively.”
Quote in the paperSubstantial claim - Findingown data
PS articles employ more adjectives than LN (75 vs. 65 per 1,000 words).
“Additionally, PS articles employ more adjectives (75 per 1,000 words) than LN (65 per 1,000 words)”
Quote in the paperSubstantial claim - Findingown data
PS relies heavily on simple syntactic structures, frequently using noun phrases (NOUN, DET) followed by prepositions (ADP) or subordinating conjunctions (SCONJ).
“In contrast, PS relies heavily on simple syntactic structures, frequently using noun phrases (NOUN, DET) followed by prepositions (ADP) or subordinating conjunctions (SCONJ).”
Quote in the paperSubstantial claim - Findingown data
PS articles have a significantly lower count of unique noun phrases compared to LN.
“Our analysis (Figure 2f) also demonstrates that PS articles have a significantly lower count of unique noun phrases compared to LN”
Quote in the paperSubstantial claim - Findingown data
PS articles frequently reuse identical templates with only superficial changes to names, locations, and statistics.
“Furthermore, as shown in figure 1, PS articles frequently reuse identical templates with only superficial changes to names, locations, and statistics, revealing a low-effort, mass-produced approach to content generation.”
Quote in the paperNot a substantial claim - Findingown data
Random Forest showed slightly better performance than XGBoost and SVM for the classification task.
“For the classification task, we use three algorithms: XGBoost, Random Forest, and SVM, with Random Forest showing slightly better performance than the others.”
Quote in the paperNot a substantial claim - Findingown data
LN articles form fewer but more dispersed clusters, reflecting greater diversity in content and style.
“In contrast, LN articles form fewer but more dispersed clusters, reflecting greater diversity in content and style without the formation of tightly packed groups.”
Quote in the paperNot a substantial claim - Interpretationown data
Categorizing Pink Slime articles as fake news may not be appropriate or accurate based on textual characteristics.
“These results suggest that, based on textual characteristics, categorizing Pink Slime articles as fake news may not be appropriate or accurate.”
Quote in the paperSubstantial claim - Interpretationreasoning
The progressive continual adaptation strategy allows the model to gradually incorporate distributional shifts induced by LLM-generated adversarial examples and retain discriminative capacity on the original PS class through rehearsal.
“This progressive continual adaptation strategy allows the model to gradually incorporate distributional shifts induced by LLM-generated adversarial examples, and retain discriminative capacity on the original PS class through rehearsal on D (t) PS-base .”
Quote in the paperSubstantial claim
- It may state two separate claims as one. 14
- Probably not a substantial claim. 5
It may state two separate claims as one.
Findingown dataFake news articles deviate significantly from PS articles, with embeddings dispersed similar to LN and national news.
“Interestingly, fake news articles deviate significantly from PS articles, with their embeddings also exhibiting a dispersed pattern similar to LN and national news”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataWhen the full set of LLM-modified samples (100%) is included, F1-score reaches 75% in the uncontrolled setting and 74% in the controlled setting.
“When the full set of LLM-modified samples (100%) is included, the F1-score reaches 75% in the uncontrolled setting and 74% in the controlled setting.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataThe uncontrolled setting incurs a modest 3% decline in F1-score on the original test distribution, whereas the controlled setting limits regression to 1%.
“The uncontrolled setting incurs a modest 3% decline in F1-score, whereas the controlled setting by leveraging rehearsal via D (t) PS-base limits the regression to just 1%.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataLN frequently uses complex POS trigrams, many top trigrams begin with AUX followed by PRON, NOUN, or PROPN.
“As shown in figure 2e, many top trigrams in LN begin with AUX, followed by other tags like PRON (Pronoun), NOUN (Noun), or PROPN (Proper Noun).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataText embedding combined with a downstream fully connected model performed better than handcrafted features but fell short of full fine-tuning.
“Additionally, a text embedding approach combined with a downstream fully connected (FC) model performed better than the handcrafted features but fell short of full fine-tuning.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataAmong open-source LLMs, Qwen was most effective at evading detection, causing a 28.34% drop in F1 against a fine-tuned BERT baseline (89%).
“As shown in table 3, among open-source models, Qwen was the most effective at evading detection: when tested against a previously fine-tuned BERT classifier (baseline F1-score: 89%), it led to a 28.34% drop in detection performance (Table 3).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataLegitimate national news demonstrates an even broader distribution, indicating highly varied content and less structural uniformity.
“Legitimate national news demonstrates an even broader distribution, indicating highly varied content and less structural uniformity.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataIncorporating 50% of LLM-modified training samples boosts F1-score by 22% in the uncontrolled setting and by 20% in the controlled setting.
“Specifically, incorporating 50% of the LLM-modified training samples boosts the F1-score by 22% in the uncontrolled setting and by 20% in the controlled setting.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataBERT shows slightly better performance (89.31% F1) compared to XLNet and Flan-T5, although the difference is not statistically significant.
“Among the models, BERT shows slightly better performance (89.31% in F1-score) compared to XLNet and Flan-T5, although the difference is not statistically significant.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationown dataThe paper presents the first comprehensive linguistic analysis of Pink Slime journalism, revealing consistent syntactic simplicity, limited lexical richness, and templated writing patterns that set it apart from legitimate local news.
“In this work, we present the first comprehensive linguistic analysis of Pink Slime journalism, revealing consistent syntactic simplicity, limited lexical richness, and templated writing patterns that set it apart from legitimate local news.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Premisenone givenNone of the aforementioned studies have conducted a comprehensive textual content analysis or developed detection strategies focused on the linguistic and stylistic properties of PS articles.
“However, none of the aforementioned studies have conducted a comprehensive textual content analysis or developed detection strategies focused on the linguistic and stylistic properties of PS articles.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductown dataThe paper's analysis was limited to a subset of 40,000 articles.
“While the full NELA-PS dataset contains over 8 million entries, our analysis was limited to a subset of 40,000 articles.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationown dataIncorporating a partial replay buffer of original PS samples effectively mitigates catastrophic forgetting and helps preserve generalization on the original distribution.
“This suggests that incorporating a partial replay buffer of original PS samples effectively mitigates catastrophic forgetting and helps preserve generalization on the original distribution.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Premisenone givenA research gap persists in leveraging the intrinsic textual properties of PS articles for robust and scalable detection methods.
“However, a research gap persists in leveraging the intrinsic textual properties of these articles for robust and scalable detection methods.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Study design and conductreasoningThe paper avoided train-test overlap by selecting entire clusters randomly to form 80% of the PS data for training, with the rest for testing, and repeated this process three times.
“Despite deduplication, many PS articles retained templated formats, so we avoided train-test overlap by selecting entire clusters-randomly chosen to form 80% of the PS data for training, with the rest for testing.”
Quote in the paperSubstantial claimProbably not a substantial claim.
Study design and conductown dataThe paper's continual learning approach incorporates original PS data as 50% of the PS sample to avoid forgetting.
“To avoid forgetting, we also incorporate original PS data, which is 50% of the PS sample.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Study design and conductnone givenThe paper did not conduct fact-checking on the datasets and relied on the provided labels as-is.
“We do not conduct any fact-checking on the datasets used in this study and relied on the provided labels as-is.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
The LLM paraphrasing strategies used in the experiments mimic strategies PS journalists might use to improve quality and evade detection.
“To simulate adversarial scenarios, we rephrased PS articles in ways that mimic strategies PS journalists might use to improve their quality and evade detection.”
The paper relies on this without support.
The labels provided in the datasets accurately reflect whether articles are PS, LN, or fake news.
“We do not conduct any fact-checking on the datasets used in this study and relied on the provided labels as-is.”
The paper relies on this without support.
The sampled subset of 40,000 articles is representative of the full NELA-PS dataset and PS journalism.
“While the full NELA-PS dataset contains over 8 million entries, our analysis was limited to a subset of 40,000 articles.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Handcrafted features are described as solely syntactic despite lexical features
MinorThe paper says handcrafted features rely solely on syntactic attributes, but its feature list explicitly includes lexical richness measures.
“In contrast, handcrafted features rely solely on syntactic attributes, making it more challenging to detect PS content effectively.”
“lexical richness Lexical richness measured by Root Type-Token Ratio (RTTR), corrected Type-Token Ratio (CTTR), and the measure of Measure of Textual Lexical Diversity (MTLD)”
BERT's accuracy is reported as its F1-score
MinorThe prose calls 89.31% BERT's F1-score, but Table 2 lists 89.31% as its accuracy and 89.05% as its F1-score.
“Among the models, BERT shows slightly better performance (89.31% in F1-score) compared to XLNet and Flan-T5, although the difference is not statistically significant.”
“fine-tuning BERT 89.31 89.05 XLNet 87.68 88.44 FlanT5 87.14 87.83”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 5
- Priyanjana Bengani. 2019. Hundreds of "pink slime" local news outlets are distributing algorithmic stories and conservative talking points. Columbia Journal- ism Review.
- Christine Sowa Lepird. 2024. Pink Slime Measuring, Finding, and Countering Online Threats to Local News. Ph.D. thesis, Syracuse University.
- Christine Sowa Lepird and M Kathleen. 2023. Au- tomated pink slime detection from social network features. In International Conference on Social Com- puting, Behavioral-Cultural Modeling and Predic- tion and Behavior Representation in Modeling and Simulation.
- PMLR.
- William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426, Vancouver, Canada. Association for Computational Linguistics. · DOI
Cited by
Sorted by citations
Pink slime journalism on Facebook: advertisement trends and political influence (2019–2024)
Lepird, C. S., Ng, L. H. X., & Carley, K. M. (2026) · Computational and Mathematical Organization Theory · doi:10.1007/s10588-026-09419-7
Metadata
- Authors
- University of Houston, T., Shahriar, S., Ayoobi, N., University of Houston, T., Mukherjee, A., University of Houston, T., Musharrat, M., University of Houston, T., Vamsi, S. V., & University of Houston, T.
- Affiliations
- University of Houston (US)
- Venue
- International Conference Recent Advances in Natural Language Processing
- Publisher
- Incoma Ltd. Shoumen, BULGARIA
- Published
- 2025
- Volume
- 2025
- Pages
- 1109-1117
- ISSN
- 1313-8502 (ISSN-L)
- 2603-2813
- OpenAlex
- W7124829562
- Semantic Scholar
- ffb93889d37cdc78ab8003a35c0430c5593a85ca
- Cited by
- 1
- References
- 0
- Citations by year
- 2026: 1
- FWCI
- 3.35
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International conference Recent advances in natural language processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Keywords
- automated news generation, local news, robust learning, large language models, stylistic analysis, lexical features
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
