The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although recent studies have found that OpenAI's GPTBot generally exhibits compliant behavior (Liu et al. 2025, Kim et al. 2025)”
From Opting Out of Generative AI? Training Data, Search Visibility, and the Traffic Rankings of News Publishers (2026) · cites this paper as Liu et al. 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Hong, J.-W., & Curran, N. M. (2019) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3326337 · Cited by 217
Research works printed but not linked: 67
- Safinah Ali and Cynthia Breazeal. 2023. Studying Artist Sentiments around AI-generated Artwork. (2023) 13 pages. arXiv:2311.13725 [cs.HC] https://arxiv.org/abs/2311.13725 · arXiv
- Yasmin AlNoamany, Michele C. Weigle, and Michael L. Nelson. 2013. Access Patterns for Robots and Humans in Web Archives. In Proc. of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries. 339-348.
- Babak Amin Azad, Oleksii Starov, Pierre Laperdrix, and Nick Nikiforakis. 2020. Web runner 2049: Evaluating third-party anti-bot services. In Proc. of 17th Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 135-159.
- Quan Bai, Gang Xiong, Yong Zhao, and Longtao He. 2014. Analysis and Detection of Bogus Behavior in Web Crawler Measurement. Procedia Computer Science, Vol. 31 (2014), 1084-1091. · doi:10.1016/j.procs.2014.05.363
- Muhammad Ahmad Bashir, Sajjad Arshad, Engin Kirda, William Robertson, and Christo Wilson. 2019. A Longitudinal Analysis of the ads.txt Standard. In Proc. ACM Internet Measurement Conference 2019. 294-307. · DOI
- Lucas Bellaiche, Rohin Shahi, Martin Harry Turpin, Anya Ragnhildstveit, Shawn Sprockett, Nathaniel Barr, Alexander Christensen, and Paul Seli. 2023. Humans versus AI: whether and why we prefer human-created compared to AI-created artwork. Cognitive Research: Principles and Implications, Vol. 8, 42 (2023), 22 pages.
- Josiah D Boucher, Gillian Smith, and Yunus Doğan Telliel. 2024. Is Resistance Futile?: Early Career Game Developers, Generative AI, and Ethical Skepticism. In Proc. of CHI Conference on Human Factors in Computing Systems 2024. 1-13.
- Gordon Burtch, Dokyun Lee, and Zhichen Chen. 2023. The consequences of generative AI for UGC and online community engagement. SSRN (2023), 26 pages.
- Eva Cetinic and James She. 2022. Understanding and Creating Art with AI: Review and Outlook. ACM Transactions on Multimedia Computing, Communications, and Applications, Vol. 18, 2 (2022), 1-22. · DOI · Paper page
- Zi Chu, Steven Gianvecchio, and Haining Wang. 2018. Bot or Human? A Behavior-Based Online Bot Detection System. From Database to Cyber Security: Essays Dedicated to Sushil Jajodia on the Occasion of His 70th Birthday (2018), 432-449.
- Michael Dinzinger and Michael Granitzer. 2024. A Longitudinal Study of Content Control Mechanisms. In Proc. of the ACM Web Conference. 1382-1387.
- Michael Dinzinger Florian Heß and Michael Granitzer. 2024. A Survey of Web Content Control for Generative AI. (2024) 12 pages. arXiv:2404.02309 [cs.IR] https://arxiv.org/abs/2404.02309 · arXiv
- Ziv Epstein, Aaron Hertzmann, Memo Akten, Hany Farid, Jessica Fjeld, Morgan R. Frank, Matthew Groh, Laura Herman, Neil Leach, Robert Mahari, Alex ''Sandy'' Pentland, Olga Russakovsky, Hope Schroeder, and Amy Smith. 2023. Art and the science of generative AI. Science, Vol. 380, 6650 (2023), 1110-1111.
- Richard Fletcher. 2024. How many news websites block AI crawlers? Reuters Institute Factsheets (2024), 7 pages.
- Manuel B. Garcia. 2024. The Paradox of Artificial Creativity: Challenges and Opportunities of Generative AI Artistry. Creativity Research Journal (2024), 1-14.
- Paolo Grigis and Antonella De Angeli. 2024. Playwriting with Large Language Models: Perceived Features, Interaction Strategies and Outcomes. In Proc. the International Conference on Advanced Visual Interfaces 2024. 1-9.
- Alicia Guo Shreya Sathyanarayanan Leijie Wang Jeffrey Heer and Amy Zhang. 2025. From Pen to Prompt: How Creative Writers Integrate AI into their Writing Practice. (2025) 23 pages. arXiv:2411.03137 [cs.HC] https://arxiv.org/abs/2411.03137 · DOI
- Eszter Hargittai. 2009. An Update on Survey Measures of Web-Oriented Digital Literacy. Social Science Computer Review, Vol. 27, 1 (2009), 130-137. · DOI · Paper page
- Xinyi Hou Yanjie Zhao and Haoyu Wang. 2024. On the (In)Security of LLM App Stores. (2024) 17 pages. arXiv:2407.08422 [cs.CR] https://arxiv.org/abs/2407.08422 · arXiv
- Hongxian Huang, Runshan Fu, and Anindya Ghose. 2023. Generative AI and Content Creators: Evidence from Digital Art Platforms. SSRN (2023), 41 pages.
- Christos Iliou, Theodoros Kostoulas, Theodora Tsikrika, Vasilis Katos, Stefanos Vrochidis, and Ioannis Kompatsiaris. 2021. Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics. Digital Threats: Research and Practice, Vol. 2, 3 (2021), 1-26. · doi:10.1145/3447815
- Christos Iliou, Theodoros Kostoulas, Theodora Tsikrika, Vasilis Katos, Stefanos Vrochidis, and Yiannis Kompatsiaris. 2019. Towards a framework for detecting advanced web bots. In Proc. of 14th International Conference on Availability, Reliability and Security. 1-10.
- Katy Ilonka Gero, Meera Desai, Carly Schnitzler, Nayun Eom, Jack Cushman, and Elena L. Glassman. 2024. Creative Writers' Attitudes on Writing as Training Data for Large Language Models. In Proc. of CHI Conference on Human Factors in Computing Systems 2025. 1-16.
- Gregoire Jacob, Engin Kirda, Christopher Kruegel, and Giovanni Vigna. 2012. PUBCRAWL: Protecting Users and Businesses from CRAWLers. In Proc. of 21st USENIX Security Symposium. 507-522.
- Steve TK Jan, Qingying Hao, Tianrui Hu, Jiameng Pu, Sonal Oswal, Gang Wang, and Bimal Viswanath. 2020. Throwing Darts in the Dark? Detecting Bots with Limited Data using Neural Data Augmentation. In Proc. of 2020 IEEE Symposium on Security and Privacy. IEEE, 1190-1206.
- Harry H Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru. 2023. AI Art and its Impact on Artists. In Proc. of AAAI ACM Conference on AI, Ethics, and Society 2023. 363-374.
- Hannah Johnston and David Thue. 2024. Understanding Visual Artists' Values and Attitudes towards Collaboration, Technology, and AI. In Proc. 50th Graphics Interface Conference. 1-9.
- Ben Jones, Tzu-Wen Lee, Nick Feamster, and Phillipa Gill. 2014. Automated Detection and Fingerprinting of Censorship Block Pages. In Proc. of ACM Internet Measurement Conference 2014. 299-304.
- Hugo Jonker, Benjamin Krumnow, and Gabry Vlot. 2019. Fingerprint Surface-Based Detection of Web Bot Detectors. In Proc. of 24th European Symposium on Research in Computer Security. Springer, 586-605.
- Reishiro Kawakami and Sukrit Venkatagiri. 2024. The Impact of Generative AI on Artists. In Proc. of 16th Conference on Creativity & Cognition. 79-82.
- Santanu Kolay, Paolo D'Alberto, Ali Dasdan, and Arnab Bhattacharjee. 2008. A Larger Scale Study of Robots.txt. In Proc. of 17th World Wide Web Conference. 1171-1172.
- Martijn Koster Gary Illyes Henner Zeller and Lizzi Sassman. 2022. Robots Exclusion Protocol. RFC 9309. https://doi.org/10.17487/RFC9309 · DOI
- Shinil Kwon, Young-Gab Kim, and Sungdeok Cha. 2012. Web robot detection based on pattern-matching technique. Journal of Information Science, Vol. 38, 2 (2012), 118-126. · doi:10.1177/0165551511435969
- Rita Latikka, Jenna Bergdahl, Nina Savela, and Atte Oksanen. 2023. AI as an Artist? A Two-Wave Survey Study on Attitudes Toward Using Artificial Intelligence in Art. Poetics, Vol. 101 (2023), 11 pages.
- Junsup Lee, Sungdeok Cha, Dongkun Lee, and Hyungkyu Lee. 2009. Classification of web robots: an empirical study based on over one billion requests. Computers & Security, Vol. 28, 8 (2009), 795-802. · DOI
- Jie Li, Hancheng Cao, Laura Lin, Youyang Hou, Ruihao Zhu, and Abdallah El Ali. 2024. User Experience Design Professionals' Perceptions of Generative Artificial Intelligence. In Proc. of CHI Conference on Human Factors in Computing Systems 2024. 1-18.
- Xigao Li, Babak Amin Azad, Amir Rahmati, and Nick Nikiforakis. 2021. Good Bot, Bad Bot: Characterizing Automated Browsing Activity. In Proc. of 2021 IEEE Symposium on Security and Privacy. IEEE, 1589-1605.
- Baker & Hostetler LLP. 2024. Case Tracker: Artificial Intelligence, Copyrights and Class Actions. https://www.bakerlaw.com/services/artificial-intelligence-ai/case-tracker-artificial-intelligence-copyrights-and-class-actions/
- Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Anis, An Dinh, Caroline Chitongo, Da Yin, Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Sandy Pentland. 2024. Consent in Crisis: The Rapid Decline of the AI Data Commons. (2024), 41 pages. arXiv:2407.14933 [cs.CL] https://arxiv.org/abs/2407.14933 · arXiv
- Anália G. Lourenço and Orlando O. Belo. 2006. Catching Web Crawlers in the Act. In Proc. of 6th International Conference on Web Engineering. 265-272.
- Juniper Lovato, Julia Witte Zimmerman, Isabelle Smith, Peter Dodds, and Jennifer L. Karson. 2024. Foregrounding Artist Opinions: A Survey Study on Transparency, Ownership, and Fairness in AI Generative Art. In Proc. of AAAI ACM Conference on AI, Ethics, and Society 2024. 905-916.
- Bron Maher. 2024. Revealed: Which of the top 100 UK and US news websites are blocking AI crawlers -- PressGazette. https://pressgazette.co.uk/platforms/news-sites-block-ai-web-crawlers-chatgpt-google/
- Elz.e Sigut.e Mikalonyt.e, and Markus Kneer. 2022. Can Artificial Intelligence Make Art?: Folk Intuitions as to whether AI-driven Robots Can Be Viewed as Artists and Produce Art. ACM Transactions on Human-Robot Interaction, Vol. 11, 4 (2022), 1-19. · DOI
- Piotr Mirowski, Juliette Love, Kory Mathewson, and Shakir Mohamed. 2024. A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians. In Proc. of the ACM Conference on Fairness, Accountability, and Transparency 2024. 1622-1636. · DOI · Paper page
- Alexis Newton and Kaustubh Dhole. 2023. Is AI Art Another Industrial Revolution in the Making? (2023), 6 pages. arXiv:2301.05133 [cs.AI] https://arxiv.org/abs/2301.05133 · arXiv
- Stack Overflow. 2024. Stack Overflow and OpenAI Partner to Strengthen the World's Most Popular Large Language Models. https://stackoverflow.co/company/press/archive/openai-partnership.
- Sungjin Park. 2024. The work of art in the age of generative AI: aura, liberation, and democratization. AI & Society (2024), 1-10.
- Kien Pham, Aécio Santos, and Juliana Freire. 2016. Understanding Website Behavior based on User Agent. In Proc. of 39th ACM SIGIR Conference on Research and Development in Information Retrieval. 1053-1056.
- Tara Poteat and Frank Li. 2021. Who You Gonna Call? An Empirical Evaluation of Website security.txt Deployment. In Proc. of ACM Internet Measurement Conference 2021. 526-532. · doi:10.1145/3487552.3487841
- Heila Precel, Allison McDonald, Brent Hecht, and Nicholas Vincent. 2024. A Canary in the AI Coal Mine: American Jews May Be Disproportionately Harmed by Intellectual Property Dispossession in Large Language Model Training. In Proc. of CHI Conference on Human Factors in Computing Systems 2024. 1-17.
- Gayatri Raman and Erin Brady. 2024. Exploring Use and Perceptions of Generative AI Art Tools by Blind Artists. (2024) 4 pages. arXiv:2409.08226 [cs.HC] https://arxiv.org/abs/2409.08226 · arXiv
- Copyright Research and Information Center. 2019. Copyright Law of Japan: Chapter II Rights of Authors (English Translation). https://www.cric.or.jp/english/clj/cl2.html.
- Stefano Rovetta, Alberto Cabri, Francesco Masulli, and Grażyna Suchacka. 2019. Bot or Not? A Case Study on Bot Recognition from Web Session Logs. Quantifying and Processing Biomedical and Behavioral Signals (2019), 197-206.
- Quirin Scheitle, Oliver Hohlfeld, Julien Gamba, Jonas Jelten, Torsten Zimmermann, Stephen D. Strowes, and Narseo Vallina-Rodriguez. 2018. A Long Way to the Top: Significance, Structure, and Stability of Internet Top Lists. In Proc. of ACM Internet Measurement Conference 2018. 478-493.
- M. H. M. Schellekens. 2013. Robot.txt: balancing interests of content producers and content users. Bridging Distances in Technology and Regulation (2013), 173-187.
- Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. 2023. Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models. In Proc. of 32nd USENIX Security Symposium. 2187-2204.
- Shawn Shan, Wenxin Ding, Josephine Passananti, Stanley Wu, Haitao Zheng, and Ben Y. Zhao. 2024. Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models. In Proc. of IEEE Symposium on Security and Privacy 2024. 807-825.
- Jingyu Shi Rahul Jain Runlin Duan and Karthik Ramani. 2023. Understanding Generative AI in Art: An Interview Study with Artists on G-AI from an HCI Perspective. (2023) 15 pages. arXiv:2310.13149 [cs.HC] https://arxiv.org/abs/2310.13149 · arXiv · doi:10.46397/jaih.15.8
- Dusan Stevanovic, Aijun An, and Natalija Vlajic. 2012. Feature evaluation for web crawler detection with data mining techniques. Expert Systems with Applications, Vol. 39, 10 (2012), 8707-8717. · doi:10.1016/j.eswa.2012.01.210
- Dongxun Su Yanjie Zhao Xinyi Hou Shenao Wang and Haoyu Wang. 2024. GPT Store Mining and Analysis. (2024) 16 pages. arXiv:2405.10210 [cs.LG] https://arxiv.org/abs/2405.10210 · arXiv
- Grażyna Suchacka, Alberto Cabri, Stefano Rovetta, and Francesco Masulli. 2021. Efficient on-the-fly Web bot detection. Knowledge-Based Systems, Vol. 223 (2021), 16 pages.
- Yang Sun, Ziming Zhuang, Isaac G. Councill, and C. Lee Giles. 2007b. Determining Bias to Search Engines from Robots.txt. In Proc. of IEEE WIC ACM Web Intelligence and Intelligent Agent Technology 2007. 149-155.
- Yang Sun, Ziming Zhuang, and C. Lee Giles. 2007a. A Large-Scale Study of Robots.txt. In Proc. of 16th World Wide Web Conference. 1123-1124.
- Antoine Vastel, Walter Rudametkin, Romain Rouvoy, and Xavier Blanc. 2020. FP-Crawlers: Studying the Resilience of Browser Fingerprinting to Block Crawlers. In Proc. of NDSS Workshop on Measurements, Attacks, and Defenses for the Web. 13 pages.
- Chyan Yang and Hsien-Jyh Liao. 2010. Using the Robots. txt and Robots Meta tags to implement online copyright and a related amendment. Library Hi Tech, Vol. 28, 1 (2010), 94-106. · doi:10.1108/07378831011026715
- Zejun Zhang Li Zhang Xin Yuan Anlan Zhang Mengwei Xu and Feng Qian. 2024. A First Look at GPT Apps: Landscape and Vulnerability. (2024) 11 pages. arXiv:2402.15105 [cs.CR] https://arxiv.org/abs/2402.15105 · arXiv
- Eric Zhou and Dokyun Lee. 2024. Generative artificial intelligence, human creativity, and art. PNAS Nexus, Vol. 3, 3 (2024), 8 pages.
Cited by
FP-Agent: Fingerprinting AI Browsing Agents
Wang, E., Shafiq, Z., & Vekaria, Y. (2026) · arXiv · doi:10.48550/arxiv.2605.01247 · Cited by 7
Whose Agent Are You? Multi-Layer Fingerprinting and Attribution of Autonomous Web Agents
Kang, D., Jeong, H., Sheffey, J., Datta, P., & Houmansadr, A. (2026) · arXiv · doi:10.48550/arxiv.2606.20910 · Cited by 3
Lin, Y.-K. (2026) · doi:10.2139/ssrn.7518878 · Cited by 1
Metadata
- Authors
- Liu, E., Luo, E., Shan, S., Voelker, G. M., Zhao, B. Y., & Savage, S.
- Affiliations
- UC San Diego, La Jolla, CA, USA
- University of Chicago, Chicago, IL, USA
- Venue
- Proceedings of the 2025 ACM Internet Measurement Conference
- Publisher
- ACM
- Published
- 2025-10-15
- Pages
- 78-99
- Cited by
- 7
- References
- 2
- Funders
- NSF (National Science Foundation) (SaTC-2241303)
- Office of Naval Research (N00014-24-1-2669)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. https://app.cochat.ai/papers/10.1145/3730567.3732913
- The paper
- Liu, E., Luo, E., Shan, S., Voelker, G. M., Zhao, B. Y., & Savage, S. (2025). Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. Proceedings of the 2025 ACM Internet Measurement Conference. https://doi.org/10.1145/3730567.3732913
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.