Strategic Response of News Publishers to Generative AI
Open accessThe DOI matches its DataCite record; 36 of the 36 references it cites were found in a registry or an index; 1 of the 12 sources its claims cite were checked for support.
Abstract
Generative AI can adversely impact news publishers by lowering consumer demand. It can also reduce demand for newsroom employees, and increase the creation of news "slop." However, it can also form a source of traffic referrals and an information-discovery channel that increases demand. We use high-frequency granular data to analyze the strategic response of news publishers to the introduction of Generative AI. Many publishers strategically blocked LLM access to their websites using the robots.txt file standard. Using a difference-in-differences approach, we find that large publishers who block GenAI bots experience reduced website traffic compared to not blocking. In addition, we find that large publishers shift toward richer content that is harder for LLMs to replicate, without increasing text volume. Finally, we find that the share of new editorial and content-production job postings rises over time. Together, these findings illustrate the levers that publishers choose to use to strategically respond to competitive Generative AI threats, and their consequences.
Overview
CoChat Summary
TL;DR
Using a high-frequency publisher panel and staggered difference-in-differences, the paper finds that blocking GenAI crawlers via robots.txt is followed by about a 7% decline in weekly visits, including in human-panel data, rather than protecting traffic. Publishers instead shift toward richer, more interactive pages without increasing text output, and editorial hiring does not contract relative to other roles.
Key findings
- Blocking adoption was widespread: about 75% of top publishers blocked an OpenAI-related crawler at some point starting mid-2023; news publishers blocked more than non-news sites, with the top 500 news sites below 60% blocking versus below 10% among top retailers.
- Blocking GenAI crawlers is followed by roughly a 7% decline in weekly visits within 6 weeks, across SimilarWeb, Semrush, and Comscore; ATT estimates are -0.074* (SimilarWeb), -0.069* (Semrush), and -0.065 (Comscore, imprecise). Negative effects are concentrated among larger publishers; lower-ranked publishers show no negative effect.
- Publishers do not scale up text: article volume decreases by 31.2%, while interactive elements increase by 68.1%, advertising/targeting technologies by 50.1%, and site framework/layout by 70.2%; multimedia increases by 40.8% but is not significant after multiple-testing adjustment. Growth is concentrated in image-related URLs.
- There is no short-term contraction in newsroom hiring relative to other roles: editorial postings do not show a discrete post-GenAI decrease, and the TWFE estimate for editorial postings after November 2022 is positive and significant (ATT = 7.578*).
Methods
- The authors construct a publisher panel linking daily traffic to robots.txt blocking rules, page structure, and hiring. Data include SimilarWeb daily domain visits (2019–Feb 2026), Semrush daily channel traffic, Comscore Web-Behavior Panel human browsing (2022–2024), HTTP Archive robots.txt and HTML snapshots, Internet Archive Wayback Machine URL counts, and Revelio job postings. The main sample is 30 NAICS 513110 newspaper publishers from 6,315 URLs matched across SimilarWeb and Revelio; Semrush/Comscore analyses expand to the top 500 news-publisher domains.
- Traffic trends are analyzed with PELT changepoint detection. The causal effect of blocking is estimated with staggered difference-in-differences following Callaway and Sant’Anna (2021) on log weekly visits, restricted to the pre-May 2024 period, with event-study windows of 12 weeks before and 6 weeks after blocking. Robustness includes Synthetic DiD, TWFE, placebo tests, selection-on-trends checks, and audits of concurrent robots.txt changes. Page-structure and hiring responses are estimated with TWFE regressions.
Limitations
- Stated: the traffic data do not capture consumption that occurs entirely within AI interfaces, and the study period precedes wider adoption of AI-integrated search products; future research should incorporate direct measures of LLM-driven referral traffic and licensing arrangements.
- Stated: Comscore estimates are imprecise, possibly due to smaller panel size; lower-traffic publisher estimates are less accurate and blocking may have smaller impact for them; job postings measure labor demand rather than realized employment and may reflect recruiting intensity or posting practices.
- Evident from the text: the main causal analysis is restricted to before May 2024, so effects after Google’s AI Overview and later algorithm updates are not causally identified; the two proposed mechanisms for traffic decline—reduced brand exposure versus lost referral clicks—are discussed but not fully separated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


















Verification
Claims the paper makes
1 Valid · 1 Not valid · 10 Not checked · 87 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“We also report ATT estimates from two other methods: Synthetic DiD (Arkhangelsky et al. 2021) and Two-Way Fixed Effects in Table 3, which yield similarly negative effects.”
What the sources say
- Supportedcites Arkhangelsky et al. 2021 matched to “Synthetic Difference-in-Differences” (2021) source
The citation is attached to the method name itself ('Synthetic DiD (Arkhangelsky et al. 2021)'), and our copy is that method's source: the Introduction (p. 2) proposes 'a new method, Synthetic Difference in Differences (SDID)' combining DID and SC, and Section 4 (p. 18) defines the target estimand as 'the average treatment effect for the treated units during the periods they were treated' — i.e., the ATT the citing sentence reports. The paper also explicitly benchmarks SDID against the two-way fixed-effects/DID regression estimator (Sections 2.2, 3, 3.1.2), so pairing SDID with TWFE is consistent with the cited work. The 'similarly negative effects' comparison is the citing authors' own Table 3 result, not an assertion attributed to Arkhangelsky et al.; in the paper's own California application (p. 8) SDID and DID indeed both give negative estimates (DID -27.3, SC -19.6, SDID smaller in magnitude). (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When we extend the analysis to smaller publishers with lower traffic in the Comscore and Semrush data, we find similarly negative but imprecisely estimated effects.”
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Many publishers strategically blocked LLM access to their websites using the robots.txt file standard.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For lower-ranked publishers (ranks 101-500), the estimate is no longer negative, which may reflect less accurate traffic estimates for lower-traffic websites and a lower impact of blocking for smaller publishers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We do not find evidence that newspapers' overall DOM elements or multimedia elements increase faster than those of retailers.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
52 more claims
Create a free account to see all 99 claims and to verify the 3 references still open.
- It may state two separate claims as one. 33
- The quote only partly states this claim. 2
- Probably not a substantial claim. 10
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1 Introduction · p. 5
“When we extend the analysis to smaller publishers with lower traffic in the Comscore and Semrush data, we find similarly negative but imprecisely estimated effects.”
§ 5.3.1 Longer time window and heterogeneity by publisher size · p. 16
“For lower-ranked publishers (ranks 101-500), the estimate is no longer negative, which may reflect less accurate traffic estimates for lower-traffic websites and a lower impact of blocking for smaller publishers.”
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Difference-in-Differences with multiple time periods
Callaway, B., & Sant’Anna, P. H. C. (2021) · Journal of Econometrics · doi:10.1016/j.jeconom.2020.12.001 · Cited by 9,126
Optimal Detection of Changepoints With a Linear Computational Cost
Killick, R., Fearnhead, P., & Eckley, I. A. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.737745 · Cited by 2,548
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150
Synthetic Difference-in-Differences
Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021) · American Economic Review · doi:10.1257/aer.20190159 · Cited by 1,518
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
An Experimental Study of the Relationship between Online Engagement and Advertising Effectiveness
Calder, B. J., Malthouse, E. C., & Schaedel, U. (2009) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2009.07.002 · Cited by 1,040
Art and the science of generative AI
Epstein, Z., Hertzmann, A., the Investigators of Human Creativity, Akten, M., Farid, H., Fjeld, J., Frank, M. R., Groh, M., Herman, L., Leach, N., et al. (2023) · Science · doi:10.1126/science.adh4451 · Cited by 788
Responses to Entry in Multi-Sided Markets: The Impact of Craigslist on Local Newspapers
Seamans, R., & Zhu, F. (2014) · Management Science · doi:10.1287/mnsc.2013.1785 · Cited by 431
The simple macroeconomics of AI
Acemoglu, D. (2025) · Economic Policy · doi:10.1093/epolic/eiae042 · Cited by 317
Chung, D. S. (2008) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2008.00414.x · Cited by 232
Ad Revenue and Content Commercialization: Evidence from Blogs
Sun, M., & Zhu, F. (2013) · Management Science · doi:10.1287/mnsc.1120.1704 · Cited by 171
Content aggregation by platforms: The case of the news media
Chiou, L., & Tucker, C. E. (2017) · Journal of Economics & Management Strategy · doi:10.1111/jems.12207 · Cited by 161
Fee or Free: When Should Firms Charge for Online Content?
Lambrecht, A., & Misra, K. (2017) · Management Science · doi:10.1287/mnsc.2015.2383 · Cited by 128
The Production of Information in an Online World
Cagé, J., Hervé, N., & Viaud, M.-L. (2015) · The Review of Economic Studies · doi:10.1093/restud/rdz061 · Cited by 122
Paywalls and the demand for news
Chiou, L., & Tucker, C. E. (2013) · Information Economics and Policy · doi:10.1016/j.infoecopol.2013.03.001 · Cited by 117
Paywalls: Monetizing Online Content
Pattabhiramaiah, A., Sriram, S., & Manchanda, P. (2018) · Journal of Marketing · doi:10.1177/0022242918815163 · Cited by 103
Spillover Effects and Freemium Strategy in Mobile App Market
Deng, Y., Lambrecht, A., & Liu, Y. (2018) · Management Science · doi:10.1287/mnsc.2022.4619 · Cited by 101
What Do News Aggregators Do? Evidence from Google News in Spain and Germany
Calzada, J., & Gil, R. (2019) · Marketing Science · doi:10.1287/mksc.2019.1150 · Cited by 85
Who is AI Replacing? The Impact of Generative AI on Online Freelancing Platforms
Demirci, O., Hannane, J., & Zhu, X. (2025) · Management Science · doi:10.1287/mnsc.2024.05420 · Cited by 84
The consequences of generative AI for online knowledge communities
Burtch, G., Lee, D., & Chen, Z. (2024) · Scientific Reports · doi:10.1038/s41598-024-61221-0 · Cited by 83 · cited 1×
“…labor displacement from GenAI despite broad task exposure (Eloundou et al. 2024, Brynjolfsson et al. 2025, Humlum and Vestergaard 2025, Demirci et al. 2025) and to work documenting reduced contributions in online knowledge communities after GenAI adoption (Burtch et al. 2024, Lyu et al. 2025).”
The Impact of Aggregators on Internet News Consumption
Athey, S. C., Möbius, M., & Pál, J. (2021) · National Bureau of Economic Research · doi:10.3386/w28746 · Cited by 80
Generative AI and Copyright: A Dynamic PerspectivePreprint
Yang, S. A., & Zhang, A. H. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4716233 · Cited by 73
The Impact of Online Competition on Local Newspapers: Evidence from the Introduction of Craigslist
Djourelova, M., Durante, R., & Martin, G. J. (2024) · The Review of Economic Studies · doi:10.1093/restud/rdae049 · Cited by 69
Who is AI Replacing? The Impact of Generative AI on Online Freelancing PlatformsPreprint
Demirci, O., Hannane, J., & Zhu, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4991774 · Cited by 36
Large Language Models, Small Labor Market Effects
Humlum, A., & Vestergaard, E. (2025) · doi:10.2139/ssrn.5219933 · Cited by 27
Copyright Policy Options for Generative Artificial Intelligence
Gans, J. S. (2024) · National Bureau of Economic Research · doi:10.3386/w32106 · Cited by 14
Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study
Kim, T., Bock, K., Luo, C., Liswood, A., Poroslay, C., & Wenger, E. (2025) · Proceedings of the 2025 ACM Internet Measurement Conference · doi:10.1145/3730567.3764471 · Cited by 11
How to Tell a (News) Story? Quantifying the Impact of News Format and Storytelling on Engagement
Ghose, D., Mudambi, S. M., Kumar, S., & Srivastava, J. (2025) · Information Systems Research · doi:10.1287/isre.2022.0650 · Cited by 6
Wikipedia Contributions in the Wake of ChatGPT
Lyu, L., Siderius, J., Li, H., Acemoğlu, D., Huttenlocher, D. P., & Ozdaglar, A. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715543 · Cited by 5
The Impact of LLM Adoption on Online User Behavior
Padilla, N., Lam, H. T., Lambrecht, A., & Hollenbeck, B. (2025) · doi:10.2139/ssrn.5393256 · Cited by 4
News Aggregators and Competition Among Newspapers in the Internet
Esfahani, N. N., & Jeon, D. (2014) · Toulouse Capitole Publications (University Toulouse 1 Capitole) · doi:10.1257/mic.20140151 · Cited by 2
How many news websites block AI crawlers?
Fletcher, R. (2024) · Reuters Institute for the Study of Journalism · doi:10.60625/risj-xm9g-ws87 · Cited by 2
Generative AI and Copyright: A Dynamic Perspective
Yang, S. A., & Zhang, A. H. (2024) · arXiv · doi:10.48550/arxiv.2402.17801
How To Tell a (News) Story? Quantifying the Impact of News Format and Storytelling on Engagement
Ghose, D., Mudambi, S. M., Kumar, S., & Srivastava, J. (2025) · doi:10.2139/ssrn.5649030
Cited by
AI and the Collapse of the www
Chan, A. (2026) · National Bureau of Economic Research · doi:10.3386/w35344 · Cited by 7
Lin, Y.-K. (2026) · doi:10.2139/ssrn.7518878 · Cited by 1
Language Industrialization: A New Social Revolution?Preprint
Geng, M. (2026) · HAL
When Search Eats the Web: A Model of Corpus Erosion under Generative ExtractionPreprint
Peyronnet, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.15896
Do Third-Party Web Traffic Estimates Preserve Causal Variation?Preprint
Khosravi, M., & Yoganarasimhan, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7514898
Metadata
- Authors
- Zhao, H., & Berman, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2512.24968
- OpenAlex
- W7118007976
- Semantic Scholar
- eb4c93223333b5a141d7045c8ed7753b39862768
- Cited by
- 6
- References
- 30
- Citations by year
- 2026: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Communication › Journalism and Media Studies
- Social Sciences › Communication › Technology's Impact on Media
- Social Sciences › Communication › News Media Framing and Political Communication
- Keywords
- generative AI, news publishers, large language models, robots.txt, website traffic, content quality, job advertisements, AI-generated content
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-17, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-17 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.