Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; none of the 32 sources its claims cite is checked for support yet.

Opting Out of Generative AI? Training Data, Search Visibility, and the Traffic Rankings of News Publishers

Open access
Lin, Yu-Kai · 2026 · SSRN Electronic Journal

Abstract

Generative AI (GenAI) products can answer users' questions with publisher content, raising concerns that they divert audiences from news websites. Publishers can opt out of AI crawlers through robots.txt, but the traffic consequences of that choice remain unclear. I distinguish between two channels through which GenAI uses publisher content: AI training, which absorbs content into a model's parameters, and AI search, which retrieves and cites content at query time. Using daily robots.txt snapshots, I identify opt-out decisions for OpenAI's GPTBot and OAI-SearchBot across 492 and 825 news websites, respectively. I then exploit GPT-4o's undisclosed knowledge cutoff and the launch of ChatGPT Search in difference-indifferences designs that account for opt-out decisions coordinated at the publisher-group level. Opting out of AI training has no detectable average effect on web traffic rankings, although it worsens the rankings of content-rich websites. Opting out of AI search worsens rankings by 4.3% on average, with effects concentrated among non-gated websites and those with limited content offerings. These findings suggest that AI training and AI search affect publishers through distinct channels, and that opt-out decisions and governance mechanisms should treat the two uses separately.

Overview

CoChat Summary

TL;DR

This paper separates generative AI’s use of news content into AI training and AI search, using robots.txt opt-outs around GPT-4o and ChatGPT Search. It finds no detectable average traffic-ranking effect of opting out of AI training, but a 4.3% average ranking deterioration from opting out of AI search.

Key findings

  • Opting out of AI training has no detectable average effect on web traffic rankings (ATT = 0.034, SE = 0.029; 95% CI −2.3% to +9.5%), although it worsens rankings for content-rich websites (ATT = 0.080, SE = 0.040; between-group difference p < 0.05).
  • Opting out of AI search worsens rankings by 4.3% on average (ATT = 0.042, SE = 0.014; 95% CI 1.5% to 7.2%), corresponding to roughly 1,428 ranking positions for the median website in the AI search sample.
  • AI search opt-out effects are concentrated among non-gated websites (ATT = 0.055, SE = 0.017) and low-richness websites (ATT = 0.079, SE = 0.026), consistent with a content-discovery channel.
  • The paper argues AI training and AI search affect publishers through distinct channels, and that opt-out decisions are often coordinated at the publisher-group level, so treating websites as independent units can distort estimates.

Methods

  • Event-study difference-in-differences using daily robots.txt snapshots from the Internet Archive; AI training panel: 492 news websites observed monthly August 2023–September 2024; AI search panel: 825 news websites observed monthly June 2024–April 2025.
  • AI training treatment: disallowed GPTBot by GPT-4o’s October 2023 knowledge cutoff versus first disallowed GPTBot November 2023–April 2024, with post period from May 2024 GPT-4o release. AI search treatment: staggered OAI-SearchBot disallow, with post period from November 2024 ChatGPT Search launch.
  • Outcome is log Tranco rank; main estimator is de Chaisemartin and D’Haultfœuille (2026); estimates use inverse publisher-size weighting and publisher-clustered standard errors, with Lighthouse metrics as controls and extensive robustness checks.

Limitations

  • The paper states that dynamic selection cannot be fully resolved: reasons for opting out remain unobservable, and time-varying confounders such as litigation, licensing negotiations, paywall changes, or broader distribution-strategy shifts cannot be ruled out.
  • The paper states that the outcome is relative web traffic ranking, not visits, so estimates cannot separate gains for opted-in sites from losses for opted-out sites or show whether total news traffic increased or decreased; rank-based SUTVA interference is a valid concern, though robustness checks suggest main conclusions are not materially affected.
  • The paper states that robots.txt measures intent-to-treat, not actual OpenAI use or compliance, and that AI search effects cannot be fully separated from the underlying pretrained model; the text also makes evident that the sample is disproportionately US-based and English-language, and the analysis cannot assess advertising revenue, subscriptions, or content outcomes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2139/ssrn.7518878

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 14 Not valid · 25 Not checked · 363 Own results and interpretations

174 more claims

Create a free account to see all 402 claims and to verify the 21 references still open.

See all claims — free account
  • The quote only partly states this claim. 3
  • It may state two separate claims as one. 166
  • Probably not a substantial claim. 102

Create a free account to see these claims.

Citations

Cites

Show all 36 references

Research works printed but not linked: 10

  1. Han M, Zheng J, Rui H (2025) Should News Organizations Share Data with Generative AI? (June 19) https://papers.ssrn.com/abstract=5313353.
  2. He S, Hollenbeck B, Proserpio D (2022) The Market for Fake Reviews. Mark. Sci. 41(5):468-493.
  3. Peukert C, Bechtold S, Batikas M, Kretschmer T (2022) Regulatory Spillovers and Data Governance: Evidence from the GDPR. Mark. Sci. 41(4):746-768.
  4. The Associated Press (2023) AP, Open AI agree to share select news content and technology in new collaboration. (July 13) https://www.ap.org/media-center/press-releases/2023/ap-open-ai-agree- to-share-select-news-content-and-technology-in-new-collaboration/.
  5. Callaway B, Sant'Anna PHC (2021) Difference-in-differences with multiple time periods. J. Econom. 225(2):200-230.
  6. de Chaisemartin C, D'Haultfœuille X (2026) Difference-in-Differences Estimators of Intertemporal Treatment Effects. Rev. Econ. Stat. 108(4):863-880.
  7. Ghanem D, Sant'Anna PHC, Wüthrich K (2026) Selection and parallel trends. (February 2) http://arxiv.org/abs/2203.09001. · arXiv
  8. Kim T, Bock K, Luo C, Liswood A, Poroslay C, Wenger E (2025) Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. Proc. 2025 ACM Internet Meas. Conf. IMC '25. (Association for Computing Machinery, New York, NY, USA), 541-557.
  9. Liu E, Luo E, Shan S, Voelker GM, Zhao BY, Savage S (2025) Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. Proc. 2025 ACM Internet Meas. Conf. IMC '25. (Association for Computing Machinery, New York, NY, USA), 78-99.
  10. Pochat VL, Goethem TV, Tajalizadehkhoob S, Korczyński M, Joosen W (2019) Tranco: a research- oriented top sites ranking hardened against manipulation. Proc. 2019 Netw. Distrib. Secur. Symp. NDSS (San Diego, CA, USA).

Metadata

Authors
Lin, Y.-K.
Affiliations
Georgia State University (US)
Corresponding author
Lin, Yu-Kai
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2026
ISSN
1556-5068 (ISSN-L)
OpenAlex
W7214430500
Cited by
1
References
36
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Locations
SSRN Electronic Journal (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
generative AI, robots.txt, AI crawlers, news publishers, AI training data, AI search, knowledge cutoff, search visibility, AI governance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-17, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-17 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.