Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record.

News Behind the News: Spillover Effects of AI Crawler Blocking Among News Publishers

Open access
Mor, Kanav, Telang, Rahul · 2026 · SSRN Electronic Journal

Abstract

The emergence of generative AI has intensified long-standing tensions between content producers and platforms over the value of online content. We examine how news organizations responded to this shift by analyzing historical robots.txt snapshots from the Internet Archive's Wayback Machine, constructing a monthly panel of news publishers across the United States, Europe, and the Rest of the World from May 2022 through December 2025. Using Interrupted Time Series and Callaway-Sant'Anna staggered difference-indifferences designs, we study publisher responses to the release of GPTBot in August 2023. We document three main findings. First, blocking accelerated after GPTBot's release across all three regions, with U.S. and European publishers increasing blocking at similar rates and more strongly than publishers in the Rest of the World. Second, paywalled outlets block more quickly and more strongly than free outlets. Third, blocking extended beyond AI-related crawlers: GPTBot blocking was followed by increased blocking of non-AI bots, with the strongest evidence of this spillover in the U.S. and Europe. These findings suggest that the rise of generative AI has contributed to a broader tightening of automated web access among news publishers, with implications extending beyond AI training to the governance of crawling on the web.

Overview

Integrity signals

Registry

Crossref · 10.2139/ssrn.6730802

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Still being checked.

Cited by

Still being checked.

Metadata

Authors
Mor, K., & Telang, R.
Affiliations
Carnegie Mellon University (US)
Corresponding author
Mor, Kanav
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2026
ISSN
1556-5068 (ISSN-L)
OpenAlex
W7171694950
Cited by
0
References
0
Type
posted content
Language
en
Open access
Locations
SSRN Electronic Journal (submittedVersion)
Topics
  • Social Sciences › Law › Intellectual Property Rights and Media
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Social Sciences › Communication › Journalism and Media Studies
Keywords
robots.txt, generative AI, news publishers, paywalls, spillover effects, Wayback Machine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.