The DOI matches its DataCite record; 30 of the 30 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Strategic Response of News Publishers to Generative AI
Open accessAbstract
Generative AI can adversely impact news publishers by lowering consumer demand. It can also reduce demand for newsroom employees, and increase the creation of news "slop." However, it can also form a source of traffic referrals and an information-discovery channel that increases demand. We use high-frequency granular data to analyze the strategic response of news publishers to the introduction of Generative AI. Many publishers strategically blocked LLM access to their websites using the robots.txt file standard. Using a difference-in-differences approach, we find that large publishers who block GenAI bots experience reduced website traffic compared to not blocking. In addition, we find that large publishers shift toward richer content that is harder for LLMs to replicate, without increasing text volume. Finally, we find that the share of new editorial and content-production job postings rises over time. Together, these findings illustrate the levers that publishers choose to use to strategically respond to competitive Generative AI threats, and their consequences.
Overview
CoChat Summary
TL;DR
Using a high-frequency publisher panel and staggered difference-in-differences, the paper finds that blocking GenAI crawlers via robots.txt is followed by about a 7% decline in weekly visits, including in human-panel data, rather than protecting traffic. Publishers instead shift toward richer, more interactive pages without increasing text output, and editorial hiring does not contract relative to other roles.
Key findings
- Blocking adoption was widespread: about 75% of top publishers blocked an OpenAI-related crawler at some point starting mid-2023; news publishers blocked more than non-news sites, with the top 500 news sites below 60% blocking versus below 10% among top retailers.
- Blocking GenAI crawlers is followed by roughly a 7% decline in weekly visits within 6 weeks, across SimilarWeb, Semrush, and Comscore; ATT estimates are -0.074* (SimilarWeb), -0.069* (Semrush), and -0.065 (Comscore, imprecise). Negative effects are concentrated among larger publishers; lower-ranked publishers show no negative effect.
- Publishers do not scale up text: article volume decreases by 31.2%, while interactive elements increase by 68.1%, advertising/targeting technologies by 50.1%, and site framework/layout by 70.2%; multimedia increases by 40.8% but is not significant after multiple-testing adjustment. Growth is concentrated in image-related URLs.
- There is no short-term contraction in newsroom hiring relative to other roles: editorial postings do not show a discrete post-GenAI decrease, and the TWFE estimate for editorial postings after November 2022 is positive and significant (ATT = 7.578*).
Methods
- The authors construct a publisher panel linking daily traffic to robots.txt blocking rules, page structure, and hiring. Data include SimilarWeb daily domain visits (2019–Feb 2026), Semrush daily channel traffic, Comscore Web-Behavior Panel human browsing (2022–2024), HTTP Archive robots.txt and HTML snapshots, Internet Archive Wayback Machine URL counts, and Revelio job postings. The main sample is 30 NAICS 513110 newspaper publishers from 6,315 URLs matched across SimilarWeb and Revelio; Semrush/Comscore analyses expand to the top 500 news-publisher domains.
- Traffic trends are analyzed with PELT changepoint detection. The causal effect of blocking is estimated with staggered difference-in-differences following Callaway and Sant’Anna (2021) on log weekly visits, restricted to the pre-May 2024 period, with event-study windows of 12 weeks before and 6 weeks after blocking. Robustness includes Synthetic DiD, TWFE, placebo tests, selection-on-trends checks, and audits of concurrent robots.txt changes. Page-structure and hiring responses are estimated with TWFE regressions.
Limitations
- Stated: the traffic data do not capture consumption that occurs entirely within AI interfaces, and the study period precedes wider adoption of AI-integrated search products; future research should incorporate direct measures of LLM-driven referral traffic and licensing arrangements.
- Stated: Comscore estimates are imprecise, possibly due to smaller panel size; lower-traffic publisher estimates are less accurate and blocking may have smaller impact for them; job postings measure labor demand rather than realized employment and may reflect recruiting intensity or posting practices.
- Evident from the text: the main causal analysis is restricted to before May 2024, so effects after Google’s AI Overview and later algorithm updates are not causally identified; the two proposed mechanisms for traffic decline—reduced brand exposure versus lost referral clicks—are discussed but not fully separated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


















Verification
Claims the paper makes
0 Valid · 1 Not valid · 11 Not checked · 87 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When we extend the analysis to smaller publishers with lower traffic in the Comscore and Semrush data, we find similarly negative but imprecisely estimated effects.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A check of a cited source is scheduled, or we don’t hold its full text yet; the claim is checked once it runs.
In the paper
“We also report ATT estimates from two other methods: Synthetic DiD (Arkhangelsky et al. 2021) and Two-Way Fixed Effects in Table 3, which yield similarly negative effects.”
What the sources say
- Not checked yetcites Arkhangelsky et al. 2021 source
A check is scheduled.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Not checked yet
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many publishers strategically blocked LLM access to their websites using the robots.txt file standard.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For lower-ranked publishers (ranks 101-500), the estimate is no longer negative, which may reflect less accurate traffic estimates for lower-traffic websites and a lower impact of blocking for smaller publishers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We do not find evidence that newspapers' overall DOM elements or multimedia elements increase faster than those of retailers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
52 more claims
Create a free account to see all 99 claims and to verify the 2 references still open.
- It may state two separate claims as one. 33
- The quote only partly states this claim. 2
- Probably not a substantial claim. 10
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1 Introduction · p. 5
“When we extend the analysis to smaller publishers with lower traffic in the Comscore and Semrush data, we find similarly negative but imprecisely estimated effects.”
§ 5.3.1 Longer time window and heterogeneity by publisher size · p. 16
“For lower-ranked publishers (ranks 101-500), the estimate is no longer negative, which may reflect less accurate traffic estimates for lower-traffic websites and a lower impact of blocking for smaller publishers.”
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126
Optimal Detection of Changepoints With a Linear Computational Cost
Killick, R., Fearnhead, P., & Eckley, I. A. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.737745 · Cited by 2,548
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150
Synthetic Difference-in-Differences
Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021) · American Economic Review · doi:10.1257/aer.20190159 · Cited by 1,518
An Experimental Study of the Relationship between Online Engagement and Advertising Effectiveness
Calder, B. J., Malthouse, E. C., & Schaedel, U. (2009) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2009.07.002 · Cited by 1,040
Responses to Entry in Multi-Sided Markets: The Impact of Craigslist on Local Newspapers
Seamans, R., & Zhu, F. (2014) · Management Science · doi:10.1287/mnsc.2013.1785 · Cited by 431
The simple macroeconomics of AI
Acemoglu, D. (2025) · Economic Policy · doi:10.1093/epolic/eiae042 · Cited by 317
Chung, D. S. (2008) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2008.00414.x · Cited by 232
Research works printed but not linked: 4
- Erik Brynjolfsson, Danielle Li, and Lindsey Raymond. Generative ai at work. The Quarterly Journal of Economics , 140(2):889-942, 2025.
- Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: Labor market impact potential of llms. Science , 384(6702):1306-1308, 2024.
- Ziv Epstein, Aaron Hertzmann, Investigators of Human Creativity, Memo Akten, Hany Farid, Jessica Fjeld, Morgan R Frank, Matthew Groh, Laura Herman, Neil Leach, et al. Art and the science of generative ai. Science , 380(6650):1110-1111, 2023.
- Nicolas Padilla, H Tai Lam, Anja Lambrecht, and Brett Hollenbeck. The impact of llm adoption on online user behavior. Available at SSRN 5393256 , 2025.
Cited by
AI and the Collapse of the www
Chan, A. (2026) · National Bureau of Economic Research · doi:10.3386/w35344 · Cited by 7
Lin, Y.-K. (2026) · doi:10.2139/ssrn.7518878 · Cited by 1
Language Industrialization: A New Social Revolution?Preprint
Geng, M. (2026) · HAL
When Search Eats the Web: A Model of Corpus Erosion under Generative ExtractionPreprint
Peyronnet, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.15896
Do Third-Party Web Traffic Estimates Preserve Causal Variation?Preprint
Khosravi, M., & Yoganarasimhan, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7514898
Metadata
- Authors
- Zhao, H., & Berman, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2512.24968
- OpenAlex
- W7118007976
- Semantic Scholar
- eb4c93223333b5a141d7045c8ed7753b39862768
- Cited by
- 6
- References
- 30
- Citations by year
- 2026: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Communication › Journalism and Media Studies
- Social Sciences › Communication › Technology's Impact on Media
- Social Sciences › Communication › News Media Framing and Political Communication
- Keywords
- generative AI, news publishers, large language models, robots.txt, website traffic, content quality, job advertisements, AI-generated content
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-17, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-17 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Strategic Response of News Publishers to Generative AI. https://app.cochat.ai/papers/10.48550/arxiv.2512.24968
- The paper
- Zhao, H., & Berman, R. (2025). Strategic Response of News Publishers to Generative AI. arXiv. https://doi.org/10.48550/arxiv.2512.24968
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.