Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its DataCite record; 30 of the 30 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.

Strategic Response of News Publishers to Generative AI

Open access
Zhao, Hangcheng, Berman, Ron · 2025 · arXiv

Abstract

Generative AI can adversely impact news publishers by lowering consumer demand. It can also reduce demand for newsroom employees, and increase the creation of news "slop." However, it can also form a source of traffic referrals and an information-discovery channel that increases demand. We use high-frequency granular data to analyze the strategic response of news publishers to the introduction of Generative AI. Many publishers strategically blocked LLM access to their websites using the robots.txt file standard. Using a difference-in-differences approach, we find that large publishers who block GenAI bots experience reduced website traffic compared to not blocking. In addition, we find that large publishers shift toward richer content that is harder for LLMs to replicate, without increasing text volume. Finally, we find that the share of new editorial and content-production job postings rises over time. Together, these findings illustrate the levers that publishers choose to use to strategically respond to competitive Generative AI threats, and their consequences.

Overview

CoChat Summary

TL;DR

Using a high-frequency publisher panel and staggered difference-in-differences, the paper finds that blocking GenAI crawlers via robots.txt is followed by about a 7% decline in weekly visits, including in human-panel data, rather than protecting traffic. Publishers instead shift toward richer, more interactive pages without increasing text output, and editorial hiring does not contract relative to other roles.

Key findings

  • Blocking adoption was widespread: about 75% of top publishers blocked an OpenAI-related crawler at some point starting mid-2023; news publishers blocked more than non-news sites, with the top 500 news sites below 60% blocking versus below 10% among top retailers.
  • Blocking GenAI crawlers is followed by roughly a 7% decline in weekly visits within 6 weeks, across SimilarWeb, Semrush, and Comscore; ATT estimates are -0.074* (SimilarWeb), -0.069* (Semrush), and -0.065 (Comscore, imprecise). Negative effects are concentrated among larger publishers; lower-ranked publishers show no negative effect.
  • Publishers do not scale up text: article volume decreases by 31.2%, while interactive elements increase by 68.1%, advertising/targeting technologies by 50.1%, and site framework/layout by 70.2%; multimedia increases by 40.8% but is not significant after multiple-testing adjustment. Growth is concentrated in image-related URLs.
  • There is no short-term contraction in newsroom hiring relative to other roles: editorial postings do not show a discrete post-GenAI decrease, and the TWFE estimate for editorial postings after November 2022 is positive and significant (ATT = 7.578*).

Methods

  • The authors construct a publisher panel linking daily traffic to robots.txt blocking rules, page structure, and hiring. Data include SimilarWeb daily domain visits (2019–Feb 2026), Semrush daily channel traffic, Comscore Web-Behavior Panel human browsing (2022–2024), HTTP Archive robots.txt and HTML snapshots, Internet Archive Wayback Machine URL counts, and Revelio job postings. The main sample is 30 NAICS 513110 newspaper publishers from 6,315 URLs matched across SimilarWeb and Revelio; Semrush/Comscore analyses expand to the top 500 news-publisher domains.
  • Traffic trends are analyzed with PELT changepoint detection. The causal effect of blocking is estimated with staggered difference-in-differences following Callaway and Sant’Anna (2021) on log weekly visits, restricted to the pre-May 2024 period, with event-study windows of 12 weeks before and 6 weeks after blocking. Robustness includes Synthetic DiD, TWFE, placebo tests, selection-on-trends checks, and audits of concurrent robots.txt changes. Page-structure and hiring responses are estimated with TWFE regressions.

Limitations

  • Stated: the traffic data do not capture consumption that occurs entirely within AI interfaces, and the study period precedes wider adoption of AI-integrated search products; future research should incorporate direct measures of LLM-driven referral traffic and licensing arrangements.
  • Stated: Comscore estimates are imprecise, possibly due to smaller panel size; lower-traffic publisher estimates are less accurate and blocking may have smaller impact for them; job postings measure labor demand rather than realized employment and may reflect recruiting intensity or posting practices.
  • Evident from the text: the main causal analysis is restricted to before May 2024, so effects after Google’s AI Overview and later algorithm updates are not causally identified; the two proposed mechanisms for traffic decline—reduced brand exposure versus lost referral clicks—are discussed but not fully separated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2512.24968

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Publishers' Daily Traffic
Figure 1 Publishers' Daily TrafficFrom the PDF
Figure 2 Change-Point Detection for Daily Traffic
Figure 2 Change-Point Detection for Daily TrafficFrom the PDF
Figure 3 Weekly Traffic by Channel (Semrush, 30 News Publishers, Worldwide, All Devices)
Figure 3 Weekly Traffic by Channel (Semrush, 30 News Publishers, Worldwide, All Devices)From the PDF
Figure 4 Fraction of websites that disallow GenAI bots
Figure 4 Fraction of websites that disallow GenAI botsFrom the PDF
Figure 5 Staggered DiD of blocking GenAI bots on publisher traffic.
Figure 5 Staggered DiD of blocking GenAI bots on publisher traffic.From the PDF
Figure 6 Possible mechanisms linking blocking to reduced publisher traffic
Figure 6 Possible mechanisms linking blocking to reduced publisher trafficFrom the PDF
Figure 7 Staggered DiD estimates for Comscore traffic by publisher size group
Figure 7 Staggered DiD estimates for Comscore traffic by publisher size groupFrom the PDF
Figure 8 Distribution of placebo estimates.
Figure 8 Distribution of placebo estimates.From the PDF
Figure 9 Aggregate element counts
Figure 9 Aggregate element countsFrom the PDF
Figure 10 Job Posting Trends for Editorial and Non-Editorial Roles
Figure 10 Job Posting Trends for Editorial and Non-Editorial RolesFrom the PDF
Image 11
From the PDF
Figure WA.2: Staggered DiD of blocking GenAI bots on publisher traffic (extended window)
Figure WA.2: Staggered DiD of blocking GenAI bots on publisher traffic (extended window)From the PDF
Figure WA.3: Site Framework and Layout
Figure WA.3: Site Framework and LayoutFrom the PDF
Figure WA.4: Article Volume
Figure WA.4: Article VolumeFrom the PDF
Figure WA.5: Interactive Features
Figure WA.5: Interactive FeaturesFrom the PDF
Figure WA.6: Visual & Media Richness
Figure WA.6: Visual & Media RichnessFrom the PDF
Figure WA.7: Ads & Targeting
Figure WA.7: Ads & TargetingFrom the PDF
Image 18
From the PDF

Verification

Claims the paper makes

0 Valid · 1 Not valid · 11 Not checked · 87 Own results and interpretations

52 more claims

Create a free account to see all 99 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 33
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 10

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 30 references

Research works printed but not linked: 4

  1. Erik Brynjolfsson, Danielle Li, and Lindsey Raymond. Generative ai at work. The Quarterly Journal of Economics , 140(2):889-942, 2025.
  2. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: Labor market impact potential of llms. Science , 384(6702):1306-1308, 2024.
  3. Ziv Epstein, Aaron Hertzmann, Investigators of Human Creativity, Memo Akten, Hany Farid, Jessica Fjeld, Morgan R Frank, Matthew Groh, Laura Herman, Neil Leach, et al. Art and the science of generative ai. Science , 380(6650):1110-1111, 2023.
  4. Nicolas Padilla, H Tai Lam, Anja Lambrecht, and Brett Hollenbeck. The impact of llm adoption on online user behavior. Available at SSRN 5393256 , 2025.

Metadata

Authors
Zhao, H., & Berman, R.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
OpenAlex
W7118007976
Cited by
6
References
30
Citations by year
2026: 6
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Communication › Journalism and Media Studies
  • Social Sciences › Communication › Technology's Impact on Media
  • Social Sciences › Communication › News Media Framing and Political Communication
Keywords
generative AI, news publishers, large language models, robots.txt, website traffic, content quality, job advertisements, AI-generated content

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-17, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-17 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Strategic Response of News Publishers to Generative AI. https://app.cochat.ai/papers/10.48550/arxiv.2512.24968
The paper
Zhao, H., & Berman, R. (2025). Strategic Response of News Publishers to Generative AI. arXiv. https://doi.org/10.48550/arxiv.2512.24968

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.