The DOI matches its DataCite record.
How many news websites block AI crawlers?
Open accessAbstract
In this factsheet we describe the proportion of news websites in ten countries that block AI (artificial intelligence) crawlers. We find that, (i) by the end 2023, 48% of the most widely used news websites across ten countries were blocking OpenAI’s (ChatGPT) crawlers. A smaller number, 24%, were blocking Google’s (Gemini) AI crawler. However, (ii) the proportion of news websites that blocked OpenAI varied considerably by country, ranging from 79% in the USA to just 20% in Mexico and Poland. For Google, the figures ranged from 60% in Germany to 7% in Poland and Spain. We also find that (iii) all types of news outlet were blocking, but the websites of legacy print publications were more likely to be blocking than those of either broadcasters or digital-born outlets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Over half of US web publishers have blocked AI crawlers from accessing their sites.
“In response, over half of US web publishers have blocked AI crawlers from accessing their sites (Fletcher 2024).”
From Bridging risk and retention: risk coping with privacy and hallucination in AI (2026) · cites this paper as Fletcher 2024
The Results (Differences by country) report that 79% of the top online news websites examined in the US blocked OpenAI’s crawlers by the end of 2023. That supports 'over half' for the sites studied, but the paper examined the 15 most widely used online news sources, not US web publishers generally. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Maher, B. 2024. ‘Politico embraces generative AI web crawlers with website redesigns’. Press Gazette, 1 February.
- Newman, N., Fletcher, R., Eddy, K., Robertson, C. T., Nielsen, R. K. 2023. Reuters Institute Digital News Report 2023. Oxford: Reuters Institute for the Study of Journalism.
- Oxford Insights. (n.d.). Government AI Readiness Index 2023. (Accessed 8 February 2024).
- Sisani, A., Sommerfeld, J. 2023. ‘Axel Springer and OpenAI partner to deepen beneficial use of AI in journalism’. Axel Springer, 13 December 13.
Cited by
Sorted by citations
Bridging risk and retention: risk coping with privacy and hallucination in AI
Shin, D., Rasul, A., & Shahzad, K. (2026) · AI & SOCIETY · doi:10.1007/s00146-026-03127-1
Metadata
- Authors
- Fletcher, R.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Fletcher, Richard
- ORCID
- Fletcher, Richard
- Venue
- Reuters Institute for the Study of Journalism
- Publisher
- Reuters Institute for the Study of Journalism
- Published
- 2024
- OpenAlex
- W6920265011
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 2
- Type
- other
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Reuters Institute for the Study of Journalism (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Keywords
- AI crawlers, news websites, website blocking, ChatGPT, Google Gemini, AI-generated content, web scraping, cross-national comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
