Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its DataCite record; 30 of the 30 references it cites were found in a registry or an index; none of the 31 sources its claims cite is checked for support yet.

ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping

Open access

Abstract

AI agents deployed on the open web are becoming more capable of performing actions that may breach contractual provisions and violate computer security laws. In this paper, we introduce ScrapeBench, the first agentic benchmark for empirically evaluating the legal compliance of AI agents with website scraping policies, namely terms of service (ToS) and robots$.$txt directives. Across widely used closed-weight and open-weight AI agents, we find that most agents exhibit low compliance rates (below 5%) when instructed to scrape websites in our sample of 1,626 popular websites that explicitly prohibit scraping. In addition, we observe that even where AI agents actively check the applicable ToS and robots$.$txt, they continue to exhibit low compliance rates. Conversely, in our sample of 269 websites that do not prohibit scraping, we observe that agents sometimes refuse to scrape despite it being lawful, such as Claude Opus 4.6 exhibiting an over-refusal rate of 98.5%. Our human baseline experiments (n = 180 participants) suggest that humans exhibit compliance rates comparable to AI agents but over-refuse at much higher rates.

Overview

CoChat Summary

TL;DR

ScrapeBench introduces an agentic benchmark for measuring whether AI agents comply with website scraping prohibitions in terms of service (ToS) and robots.txt across 1,895 popular websites. It finds that most agents violate scraping prohibitions under neutral prompts, compliance rises mainly when full legal texts are pasted into prompts, and some agents over-refuse on sites that do not prohibit scraping.

Key findings

  • On 1,626 EXPLICITLY PROHIBITED websites, neutral P1 compliance was near zero for GPT, Claude, Gemini, Kimi, and Qwen, while Grok reached 11%; implicit mention (P2) did not improve compliance (highest Grok/Kimi at 5.5%–5.7%). P5 pasted ToS/robots.txt pushed GPT, Claude, and Gemini above 70% compliance, while Qwen remained at 2.9%; P7 strong legal warning yielded the highest rates for five of six agents: Claude 85.9%, Gemini 85.7%, Kimi 62.7%, Grok 59.7%, Qwen 13.8%, while GPT peaked at P5 (73.9%) and dropped at P7 (60.6%).
  • Under adversarial P8, most agents showed low compliance: Gemini 41.6%, GPT 11.7%, Kimi 9.5%, Qwen 1.7%, Grok 0.6%; Claude reached 99.7% but manual inspection suggested blanket refusal rather than reasoning about the specific site.
  • On 269 NOT EXPLICITLY PROHIBITED websites, Claude Opus 4.6 exhibited an over-refusal rate of 98.5% (abstract); in-context legal resources reduced over-refusal for Claude from 16.4% to 8.2%/6.7% (P3/P5/P6) and for Gemini from 12.6% to 1.5% (P3/P5), while GPT, Qwen, and Grok stayed below 6%.
  • Checking legal resources did not ensure compliance: all six agents retrieved robots.txt at ≥60%, with GPT, Claude, Gemini, and Grok near 99%–100%, yet P4 compliance remained low (GPT 14.2%, Claude 41.5%, Gemini 16.6%, Kimi 4.7%, Qwen 7.6%, Grok 7.9%); GPT retrieved ToS 82.8% and robots.txt 100% but complied only 14.2%. Rationales for violations were most often “robots.txt permits scraping; ToS disregarded” (69.7%) and “ToS misread as permitting scraping” (18.5%), together ~88%. Human baseline: P1 compliance 48.9% on EP sites and over-refusal 38.9% on NEP sites; P7 compliance rose to 65.1% (p < 0.001), while over-refusal rose to 45.8% (not significant).

Methods

  • Dataset: top 3,000 global websites from the Chrome Top Million Websites (CrUX) repository for February 2026, filtered to 1,895 English, US-accessible, non-NSFW sites; 1,626 labeled EXPLICITLY PROHIBITED and 269 NOT EXPLICITLY PROHIBITED. ToS and robots.txt were collected with a Python BeautifulSoup scraper; GPT-5.2 extracted verbatim scraping-relevant snippets and assigned permissibility labels, with a 15% manual quality check showing 97.9% ToS and 98.9% robots.txt agreement (Cohen’s κ = 0.91).
  • Agentic evaluation: six models (GPT-5.2, Claude Opus 4.6, Gemini 3.1 Pro, Kimi K2.5, Qwen3 Max, Grok 4.20 Beta) accessed via OpenRouter; each trial used a fetch_url tool up to K = 5 times under eight legal priming prompts (P1–P8), with temperature 0.0 and max 4,096 output tokens. GPT-5.2 served as evaluator to classify scrape actions and outcomes (violation, compliance, permissible, over-refusal), and tool traces recorded whether agents checked robots.txt or ToS pages.
  • Human baseline: 180 Prolific participants (30 per condition) in a between-subjects design across P1/P4/P7 × EP/NEP; each decided whether to scrape 15 randomly sampled websites through a simulated web app. Analyses included χ² tests, and violation rationales were categorized using GPT-5.4-Mini as a judge over 1,260 violation trials.

Limitations

  • Stated: the benchmark is a February 2026 CrUX snapshot filtered to high-traffic, English, US-accessible websites; ToS and robots.txt may change, so re-validation is encouraged, and less-recognized/non-English sites are left for future work. The setup is single-agent, single-trial, single-turn, with no multi-turn, multi-trial, or multi-agent decomposition, and the authors do not claim generalization to open-ended workflows. Legal framing notes that enforceability varies by jurisdiction and that the study does not claim any specific action is or is not a legal violation; some models accessed sites via third-party readers/proxies (GPT-5.2 for 21.6% of website data, with 2.6% initially blocked), treated the same as direct access.
  • Stated: websites with partial or conditional scraping restrictions were removed to preserve a strict binary EP/NEP label, and the human study simulated scraping with pre-saved HTML rather than live scraping.
  • Evident: the evaluation covers only six models and eight prompt conditions, with P5–P7 requiring users to paste full legal texts; labeling and rationale classification rely on LLM judges (GPT-5.2 and GPT-5.4-Mini), and the human baseline tests only P1/P4/P7 with 30 participants per condition.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2611.11306

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Task Overview. Most AI agents tend to violate the legal policies of popular websites when instructed to scrape their landing page. Inserting into the prompt a strong legal warning and the text of the website's terms of service (ToS) and robots.txt generally results in higher levels of compliance, as schematically illustrated in this figure.
Figure 1 Task Overview. Most AI agents tend to violate the legal policies of popular websites when instructed to scrape their landing page. Inserting into the prompt a strong legal warning and the text of the website's terms of service (ToS) and robots.txt generally results in higher levels of compliance, as schematically illustrated in this figure.CC-BY-SA · From the PDF
Figure 2 Breakdown of website data for SCRAPEBENCH ( n total = 1,895, n EP = 1,626, n NEP = 269) over domain categories and scraping permissibility.
Figure 2 Breakdown of website data for SCRAPEBENCH ( n total = 1,895, n EP = 1,626, n NEP = 269) over domain categories and scraping permissibility.CC-BY-SA · From the PDF
Figure 3 Compliance rates of AI agents on EXPLICITLY PROHIBITED websites ( n = 1,626) across eight legal priming prompts.
Figure 3 Compliance rates of AI agents on EXPLICITLY PROHIBITED websites ( n = 1,626) across eight legal priming prompts.CC-BY-SA · From the PDF
Figure 4 Over-refusal rates of AI agents on NOT EXPLICITLY PROHIBITED websites ( n = 269) across eight legal priming prompts.
Figure 4 Over-refusal rates of AI agents on NOT EXPLICITLY PROHIBITED websites ( n = 269) across eight legal priming prompts.CC-BY-SA · From the PDF
Figure 5 ToS and robots.txt checking rates for AI agents and humans across select legal priming prompts (P1, P4, P7) for the EXPLICITLY PROHIBITED (EP) websites.
Figure 5 ToS and robots.txt checking rates for AI agents and humans across select legal priming prompts (P1, P4, P7) for the EXPLICITLY PROHIBITED (EP) websites.CC-BY-SA · From the PDF
Figure 6 ToS and robots.txt checking rates for AI agents and humans across select legal priming prompts (P1, P4, P7) for the NOT EXPLICITLY PROHIBITED (NEP) websites.
Figure 6 ToS and robots.txt checking rates for AI agents and humans across select legal priming prompts (P1, P4, P7) for the NOT EXPLICITLY PROHIBITED (NEP) websites.CC-BY-SA · From the PDF
Figure 7 Consent form of the study with full details of the authors, purpose, procedure, confidentiality, contact information, and compensation displayed through the web app. Institutional contact details have been redacted.
Figure 7 Consent form of the study with full details of the authors, purpose, procedure, confidentiality, contact information, and compensation displayed through the web app. Institutional contact details have been redacted.CC-BY-SA · From the PDF
Figure 8 Introductory task description and instructions for participants assigned to P1 (neutral prompt).
Figure 8 Introductory task description and instructions for participants assigned to P1 (neutral prompt).CC-BY-SA · From the PDF
Figure 9 Sample presentation of website for participants assigned to P1 (neutral prompt) with the following elements: 1) a question to elicit understanding of what the website is about and 2) two panels for the decision on whether to scrape or not to scrape.
Figure 9 Sample presentation of website for participants assigned to P1 (neutral prompt) with the following elements: 1) a question to elicit understanding of what the website is about and 2) two panels for the decision on whether to scrape or not to scrape.CC-BY-SA · From the PDF
Figure 10 Introductory task description and instructions for participants assigned to P4 (resource linked).
Figure 10 Introductory task description and instructions for participants assigned to P4 (resource linked).CC-BY-SA · From the PDF
Figure 11 Sample presentation of website for participants assigned to P4 (resource linked) with the following elements: 1) a question to elicit understanding of what the website is about; 2) clickable buttons to view the website's ToS and robots.txt , and 3) two panels for the decision on whether to scrape or not to scrape.
Figure 11 Sample presentation of website for participants assigned to P4 (resource linked) with the following elements: 1) a question to elicit understanding of what the website is about; 2) clickable buttons to view the website's ToS and robots.txt , and 3) two panels for the decision on whether to scrape or not to scrape.CC-BY-SA · From the PDF
Figure 12 Introductory task description and instructions for participants assigned to P7 (strong legal warning).
Figure 12 Introductory task description and instructions for participants assigned to P7 (strong legal warning).CC-BY-SA · From the PDF
Figure 13 Sample presentation of website for participants assigned to P7 (strong legal warning) with the following elements: 1) a question to elicit understanding of what the website is about, 2) a text panel with legal warning mentioning potential violation of CFAA for non-compliance of ToS, 3) clickable buttons to view the website's ToS and robots.txt , and 4) two panels for the decision on whether to scrape or not to scrape.
Figure 13 Sample presentation of website for participants assigned to P7 (strong legal warning) with the following elements: 1) a question to elicit understanding of what the website is about, 2) a text panel with legal warning mentioning potential violation of CFAA for non-compliance of ToS, 3) clickable buttons to view the website's ToS and robots.txt , and 4) two panels for the decision on whether to scrape or not to scrape.CC-BY-SA · From the PDF
Figure 14 Sample presentation if the participant decided to scrape the website. The HTML landing pages of the websites used for the study have already been manually saved beforehand and presented to the participant, thereby simulating a scrape on their perspective.
Figure 14 Sample presentation if the participant decided to scrape the website. The HTML landing pages of the websites used for the study have already been manually saved beforehand and presented to the participant, thereby simulating a scrape on their perspective.CC-BY-SA · From the PDF
Figure 15 Sample presentation of the panel shown when a participant clicks the button for viewing the terms of service (ToS) of a website before deciding to scrape or not.
Figure 15 Sample presentation of the panel shown when a participant clicks the button for viewing the terms of service (ToS) of a website before deciding to scrape or not.CC-BY-SA · From the PDF
Figure 16 Sample presentation of the panel shown when a participant clicks the button for viewing the robots.txt of a website before deciding to scrape or not.
Figure 16 Sample presentation of the panel shown when a participant clicks the button for viewing the robots.txt of a website before deciding to scrape or not.CC-BY-SA · From the PDF
Figure 17 An optional feedback request is presented after the participant finishes the entire task of 15 websites.
Figure 17 An optional feedback request is presented after the participant finishes the entire task of 15 websites.CC-BY-SA · From the PDF
Figure 18 Debriefing form of the study for the participants after completing the task with full disclosure of the intent of the study. A disclaimer is presented noting that the participants did not actually conduct legitimate scraping by clicking the scrape button. Details that can violate anonymity have been redacted.
Figure 18 Debriefing form of the study for the participants after completing the task with full disclosure of the intent of the study. A disclaimer is presented noting that the participants did not actually conduct legitimate scraping by clicking the scrape button. Details that can violate anonymity have been redacted.CC-BY-SA · From the PDF

Verification

Claims the paper makes

0 Valid · 0 Not valid · 10 Not checked · 118 Own results and interpretations

29 more claims

Create a free account to see all 128 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 68
  • Probably not a substantial claim. 53
  • The quote only partly states this claim. 7

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 10 references

Research works printed but not linked: 3

  1. S. Longpre, R. Mahari, A. Lee, C. Lund, H. Oderinwale, W. Brannon, N. Saxena, N. Obeng- Marnu, T. South, C. Hunter, K. Klyman, C. Klamm, H. Schoelkopf, N. Singh, M. Cherep, A. M. Anis, A. Dinh, C. Chitongo, D. Yin, D. Sileo, D. Mataciunas, D. Misra, E. Alghamdi, E. Shippole, J. Zhang, J. Materzynska, K. Qian, K. Tiwary, L. Miranda, M. Dey, M. Liang, M. Hamdy, N. Muennighoff, S. Ye, S. Kim, S. Mohanty, V. Gupta, V. Sharma, V. M. Chien, X. Zhou, Y. Li, C. Xiong, L. Villa, S. Biderman, H. Li, D. Ippolito, S. Hooker, J. Kabbara, and S. Pentland. Consent in Crisis: The Rapid Decline of the AI Data Commons. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 108042-108087.
  2. A. Reuel, A. Hardy, C. Smith, M. Lamparth, M. Hardy, and M. J. Kochenderfer. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 21763-21813.
  3. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. In International Conference on Learning Representations, volume 2024, pages 15585-15606, 2024.

Cited by

See all 2

Metadata

Authors
Imperial, J. M., Slate, D., & Kolt, N.
Affiliations
  • University of Notre Dame (US)
  • Hebrew University of Jerusalem (IL)
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
OpenAlex
W7225716517
Cited by
2
References
10
Citations by year
2026: 2
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by-sa)
Funders
Hebrew University of Jerusalem
Topics
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Artificial Intelligence in Law
Keywords
AI agents, legal compliance, terms of use, robots.txt, benchmark, over-refusal, Claude Opus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping. https://app.cochat.ai/papers/10.48550/arxiv.2611.11306
The paper
Imperial, J. M., Slate, D., & Kolt, N. (2026). ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping. arXiv. https://doi.org/10.48550/arxiv.2611.11306

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.