Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 11

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study

Kim, Taein, Bock, Karstan, Luo, Claire, Liswood, Amanda, Poroslay, Chloe, Wenger, Emily · 2025 · Proceedings of the 2025 ACM Internet Measurement Conference

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3730567.3764471

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 4 references

Research works printed but not linked: 25

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al . 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 (2024). · arXiv
  2. Babak Amin Azad, Oleksii Starov, Pierre Laperdrix, and Nick Nikiforakis. 2020. Web runner 2049: Evaluating third-party anti-bot services. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 135--159.
  3. Elie Bursztein, Jonathan Aigrain, Angelika Moscicki, and John C Mitchell. 2014. The end is nigh: Generic solving of text-based {CAPTCHAs}. In 8th USENIX Workshop on Offensive Technologies (WOOT 14).
  4. Elie Bursztein, Steven Bethard, Celine Fabry, John C Mitchell, and Dan Jurafsky. 2010. How good are humans at solving CAPTCHAs? A large scale evaluation. In IEEE Security & Privacy (SP).
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1--113.
  6. Hoang Dai Nguyen, Karthika Subramani, Bhupendra Acharya, Roberto Perdisci, and Phani Vadrevu. 2024. C-Frame: Characterizing and measuring in-the-wild CAPTCHA attacks. In IEEE Security & Privacy (SP).
  7. Michael Dinzinger and Michael Granitzer. 2024. A longitudinal study of content control mechanisms. In Companion Proceedings of the ACM Web Conference 2024. 1382--1387.
  8. Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). · arXiv
  9. Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-based Software Architectures. https://ics.uci.edu/~fielding/pubs/dissertation/top.htm?deviceId=c9100f78--4e17--449f-b247--5f55bf2b13bc.
  10. Meriem Guerar, Luca Verderame, Mauro Migliardi, Francesco Palmieri, and Alessio Merlo. 2021. Gotta captcha em all: A survey of 20 years of the human-or-computer dilemma. ACM Computing Surveys (CSUR) (2021).
  11. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In Proc. of ICML.
  12. Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282 (2020). · arXiv
  13. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys 55, 12 (2023).
  14. Martijn Koster. 1994. Important: Spiders, Robots and Web Wanderers. https://web.archive.org/web/20131029200350/http://inkdroid.org/tmp/www-talk/4113.html.
  15. Junsup Lee, Sungdeok Cha, Dongkun Lee, and Hyungkyu Lee. 2009. Classification of web robots: an empirical study based on over one billion requests. Computers & Security 28, 8 (2009).
  16. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Proc. of NeurIPS (2020).
  17. Enze Liu, Elisa Luo, Shawn Shan, Geoffrey M Voelker, Ben Y Zhao, and Stefan Savage. 2024. Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. arXiv preprint arXiv:2411.15091 (2024). · arXiv
  18. Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. 2025. Consent in crisis: the rapid decline of the AI data commons. Proc. of NeurIPS (2025).
  19. 10.1145/2911451.2914757 · DOI · Paper page
  20. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al . 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Proc. of NeurIPS (2022).
  21. Yang Sun, Ziming Zhuang, and C Lee Giles. 2007. A large-scale study of robots. txt. In Proceedings of the 16th international conference on World Wide Web. 1123--1124.
  22. Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al . 2024. Jamba-1.5: Hybrid Transformer-Mamba Models at Scale. arXiv preprint arXiv:2408.12570 (2024). · arXiv
  23. 10.5555/1766171.1766196 · doi:10.5555/1766171.1766196
  24. Luis Von Ahn, Benjamin Maurer, Colin McMillen, David Abraham, and Manuel Blum. 2008. Recaptcha: Human-based character recognition via web security measures. Science 321, 5895 (2008).
  25. Guixin Ye, Zhanyong Tang, Dingyi Fang, Zhanxing Zhu, Yansong Feng, Pengfei Xu, Xiaojiang Chen, and Zheng Wang. 2018. Yet another text captcha solver: A generative adversarial network based approach. In ACM Conference on Computer and Communications Security (CCS).

Metadata

Authors
Kim, T., Bock, K., Luo, C., Liswood, A., Poroslay, C., & Wenger, E.
Affiliations
  • Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
  • Office of Information Technology, Duke University, Durham, NC, USA
Venue
Proceedings of the 2025 ACM Internet Measurement Conference
Publisher
ACM
Published
2025-10-28
Pages
541-557
Cited by
11
References
4

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. https://app.cochat.ai/papers/10.1145/3730567.3764471
The paper
Kim, T., Bock, K., Luo, C., Liswood, A., Poroslay, C., & Wenger, E. (2025). Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. Proceedings of the 2025 ACM Internet Measurement Conference. https://doi.org/10.1145/3730567.3764471

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.