Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Chatbots are used in many applications, e.g., automated agents, smart home assistants, interactive characters in online games, etc. Therefore, it is crucial to ensure they do not behave in undesired manners, providing offensive or toxic responses to users. This is not a trivial task as state-of-the-art chatbot models are trained on large, public datasets openly collected from the Internet. This paper presents a first-of-its-kind, large-scale measurement of toxicity in chatbots. We show that publicly available chatbots are prone to providing toxic responses when fed toxic queries. Even more worryingly, some non-toxic queries can trigger toxic responses too. We then set out to design and experiment with an attack, ToxicBuddy, which relies on fine-tuning GPT-2 to generate non-toxic queries that make chatbots respond in a toxic manner. Our extensive experimental evaluation demonstrates that our attack is effective against public chatbot models and outperforms manually-crafted malicious queries proposed by previous work. We also evaluate three defense mechanisms against ToxicBuddy, showing that they either reduce the attack performance at the cost of affecting the chatbot's utility or are only effective at mitigating a portion of the attack. This highlights the need for more research from the computer security and online safety communities to ensure that chatbot models do not hurt their users. Overall, we are confident that ToxicBuddy can be used as an auditing tool and that our work will pave the way toward designing more effective defenses for chatbot safety.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“, vis-à-vis attacks like model extraction [61], membership inference attacks [55], and adversarial examples [41].”
BackgroundThe Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.30 · Cited by 1,076
Self-citationBaumgartner, J., Zannettou, S., Keegan, B., Squire, M., & Blackburn, J. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7347 · Cited by 881
Self-citation
Research works printed but not linked: 53
- Yoshua Bengio , Réjean Ducharme , and Pascal Vincent. A Neural Probabilistic Language Model. In Annual Conference on Neural Information Processing Systems (NIPS) , pages 932 -- 938 . NIPS, 2000 . Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A Neural Probabilistic Language Model. In Annual Conference on Neural Information Processing Systems (NIPS), pages 932--938. NIPS, 2000.
- Antoine Bordes , Y- Lan Boureau , and Jason Weston. Learning End-to-End Goal- Oriented Dialog. In International Conference on Learning Representations (ICLR) , 2017 . Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learning End-to-End Goal- Oriented Dialog. In International Conference on Learning Representations (ICLR), 2017.
- Mohit Chandra , Dheeraj Reddy Pailla , Himanshu Bhatia , AadilMehdi J. Sanchawala , Manish Gupta , Manish Shrivastava , and Ponnurangam Kumaraguru . ? Subverting the Jewtocracy" : Online Antisemitism Detection Using Multimodal Deep Learning . In ACM Conference on Web Science (WebSci) , pages 148 -- 157 . ACM, 2021 . Mohit Chandra, Dheeraj Reddy Pailla, Himanshu Bhatia, AadilMehdi J. Sanchawala, Manish Gupta, Manish Shrivastava, and Ponnurangam Kumaraguru. ?Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learning. In ACM Conference on Web Science (WebSci), pages 148--157. ACM, 2021.
- 10.1145/3465336.3475111 · DOI
- Despoina Chatzakou , Ilias Leontiadis , Jeremy Blackburn , Emiliano De Cristofaro , Gianluca Stringhini, Athena Vakali, and Nicolas Kourtellis. Detecting Cyberbullying and Cyberaggression in Social Media. CoRR abs/1907.08873 , 2019 . Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Athena Vakali, and Nicolas Kourtellis. Detecting Cyberbullying and Cyberaggression in Social Media. CoRR abs/1907.08873, 2019. · arXiv
- 10.18653/v1/2020.emnlp-main.656 · DOI · Paper page
- Markus Freitag and Yaser Al-Onaizan . Beam Search Strategies for Neural Machine Translation. CoRR abs/1702.01806 , 2017 . Markus Freitag and Yaser Al-Onaizan. Beam Search Strategies for Neural Machine Translation. CoRR abs/1702.01806, 2017. · arXiv
- Jianfeng Gao , Michel Galley , and Lihong Li . Neural Approaches to Conversational AI . In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 2 -- 7 . ACL, 2018 . Jianfeng Gao, Michel Galley, and Lihong Li. Neural Approaches to Conversational AI. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2--7. ACL, 2018.
- Tianxing He and James R. Glass. Detecting Egregious Responses in Neural Sequence-to-sequence Models. In International Conference on Learning Representations (ICLR) , 2019 . Tianxing He and James R. Glass. Detecting Egregious Responses in Neural Sequence-to-sequence Models. In International Conference on Learning Representations (ICLR), 2019.
- Gabriel Emile Hine , Jeremiah Onaolapo , Emiliano De Cristofaro , Nicolas Kourtel- lis, Ilias Leontiadis , Riginos Samaras , Gianluca Stringhini , and Jeremy Blackburn . Kek, Cucks, and God Emperor Trump : A Measurement Study of 4chan's Politi- cally Incorrect Forum and Its Effects on the Web . In International Conference on Web and Social Media (ICWSM) , pages 92 -- 101 . AAAI, 2017 . Gabriel Emile Hine, Jeremiah Onaolapo, Emiliano De Cristofaro, Nicolas Kourtel- lis, Ilias Leontiadis, Riginos Samaras, Gianluca Stringhini, and Jeremy Blackburn. Kek, Cucks, and God Emperor Trump: A Measurement Study of 4chan's Politi- cally Incorrect Forum and Its Effects on the Web. In International Conference on Web and Social Media (ICWSM), pages 92--101. AAAI, 2017.
- Geoffrey E. Hinton , Oriol Vinyals , and Jeffrey Dean . Distilling the Knowledge in a Neural Network. CoRR abs/1503.02531 , 2015 . Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. CoRR abs/1503.02531, 2015. · arXiv
- Hossein Hosseini , Sreeram Kannan , Baosen Zhang , and Radha Poovendran . Deceiving Google's Perspective API Built for Detecting Toxic Comments. CoRR abs/1702.08138 , 2017 . Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. Deceiving Google's Perspective API Built for Detecting Toxic Comments. CoRR abs/1702.08138, 2017. · arXiv
- Chandra Khatri , Behnam Hedayatnia , Anu Venkatesh , Jeff Nunn , Yi Pan , Qing Liu , Han Song , Anna Gottardi , Sanjeev Kwatra , Sanju Pancholi , Ming Cheng , Qinglang Chen , Lauren Stubel , Karthik Gopalakrishnan , Kate Bland , Raefer Gabriel , Arindam Mandal , Dilek Hakkani-Tür , Gene Hwang , Nate Michel , Eric King , and Rohit Prasad . Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize. CoRR abs/1812.10757 , 2018 . Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, Ming Cheng, Qinglang Chen, Lauren Stubel, Karthik Gopalakrishnan, Kate Bland, Raefer Gabriel, Arindam Mandal, Dilek Hakkani-Tür, Gene Hwang, Nate Michel, Eric King, and Rohit Prasad. Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize. CoRR abs/1812.10757, 2018. · arXiv
- Virapat Kieuvongngam , Bowen Tan , and Yiming Niu . Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2. CoRR abs/2006.01997 , 2020 . Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2. CoRR abs/2006.01997, 2020. · arXiv
- 10.18653/v1/N16-1014 · DOI · Paper page
- Jiwei Li , Will Monroe , and Dan Jurafsky . A Simple , Fast Diverse Decoding Algorithm for Neural Generation. CoRR abs/1611.08562 , 2016 . Jiwei Li, Will Monroe, and Dan Jurafsky. A Simple, Fast Diverse Decoding Algorithm for Neural Generation. CoRR abs/1611.08562, 2016. · arXiv
- Jiwei Li , Will Monroe , Alan Ritter , Dan Jurafsky , Michel Galley , and Jianfeng Gao. Deep Reinforcement Learning for Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1192 -- 1202 . ACL, 2016 . Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep Reinforcement Learning for Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1192--1202. ACL, 2016.
- Haochen Liu , Tyler Derr , Zitao Liu , and Jiliang Tang . Say What I Want: Towards the Dark Side of Neural Dialogue Models. CoRR abs/1909.06044 , 2019 . Haochen Liu, Tyler Derr, Zitao Liu, and Jiliang Tang. Say What I Want: Towards the Dark Side of Neural Dialogue Models. CoRR abs/1909.06044, 2019. · arXiv
- 10.18653/v1/2022.naacl-main.204 · DOI
- Alexander H. Miller , Will Feng , Adam Fisch , Jiasen Lu , Dhruv Batra , Antoine Bordes , Devi Parikh , and Jason Weston . ParlAI: A Dialog Research Software Platform. CoRR abs/1705.06476 , 2017 . Alexander H. Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. ParlAI: A Dialog Research Software Platform. CoRR abs/1705.06476, 2017. · arXiv
- Alexandros Mittos , Savvas Zannettou , Jeremy Blackburn , and Emiliano De Cristo- faro. " And We Will Fight For Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In International Conference on Web and Social Media (ICWSM) , pages 452 -- 463 . AAAI, 2020 . Alexandros Mittos, Savvas Zannettou, Jeremy Blackburn, and Emiliano De Cristo- faro. "And We Will Fight For Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In International Conference on Web and Social Media (ICWSM), pages 452--463. AAAI, 2020.
- Abby Ohlheiser . Trolls turned Tay , Microsoft's fun millennial AI bot, into a genocidal maniac. https://wapo.st/3mRIOww , 2016 . Abby Ohlheiser. Trolls turned Tay, Microsoft's fun millennial AI bot, into a genocidal maniac. https://wapo.st/3mRIOww, 2016.
- Nedjma Ousidhoum , Xinran Zhao , Tianqing Fang , Yangqiu Song , and Dit-Yan Yeung. Probing Toxic Content in Large Pre-Trained Language Models. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 4262 -- 4274 . ACL, 2021 . Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing Toxic Content in Large Pre-Trained Language Models. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4262--4274. ACL, 2021.
- 10.1609/icwsm.v16i1.19330 · DOI
- 10.1609/icwsm.v14i1.7354 · DOI
- 10.1109/SP.2016.41 · DOI · Paper page
- Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . Bleu : a Method for Automatic Evaluation of Machine Translation . In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 311 -- 318 . ACL, 2016 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a Method for Automatic Evaluation of Machine Translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 311--318. ACL, 2016.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners. OpenAI blog , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI blog, 2019.
- Ashwin Rajadesingan , Paul Resnick , and Ceren Budak . Quick , Community-Specific Learning: How Distinctive Toxicity Norms Are Maintained in Political Subreddits. In International Conference on Web and Social Media (ICWSM) , pages 557 -- 568 . AAAI, 2020 . Ashwin Rajadesingan, Paul Resnick, and Ceren Budak. Quick, Community-Specific Learning: How Distinctive Toxicity Norms Are Maintained in Political Subreddits. In International Conference on Web and Social Media (ICWSM), pages 557--568. AAAI, 2020.
- Manoel Horta Ribeiro , Jeremy Blackburn , Barry Bradlyn , Emiliano De Cristofaro , Gianluca Stringhini , Summer Long , Stephanie Greenberg , and Savvas Zannettou . The Evolution of the Manosphere across the Web . In International Conference on Web and Social Media (ICWSM) , pages 196 -- 207 . AAAI, 2021 . Manoel Horta Ribeiro, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro, Gianluca Stringhini, Summer Long, Stephanie Greenberg, and Savvas Zannettou. The Evolution of the Manosphere across the Web. In International Conference on Web and Social Media (ICWSM), pages 196--207. AAAI, 2021.
- Alan Ritter , Colin Cherry , and Bill Dolan. Unsupervised Modeling of Twitter Conversations. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages 172 -- 180 . ACL, 2010 . Alan Ritter, Colin Cherry, and Bill Dolan. Unsupervised Modeling of Twitter Conversations. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 172--180. ACL, 2010.
- Sascha Rothe , Shashi Narayan , and Aliaksei Severyn . Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. CoRR abs/1907.12461 , 2019 . Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. CoRR abs/1907.12461, 2019. · arXiv
- Maarten Sap , Dallas Card , Saadia Gabriel , Yejin Choi , and Noah A . Smith. The Risk of Racial Bias in Hate Speech Detection. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 1668 -- 1678 . ACL, 2019 . Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. The Risk of Racial Bias in Hate Speech Detection. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1668--1678. ACL, 2019.
- Emily Sheng , Kai-Wei Chang , Premkumar Natarajan , and Nanyun Peng . Towards Controllable Biases in Language Generation. CoRR abs/2005.00268 , 2020 . Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. Towards Controllable Biases in Language Generation. CoRR abs/2005.00268, 2020. · arXiv
- Kurt Shuster , Da Ju , Stephen Roller , Emily Dinan , Y- Lan Boureau , and Jason We- ston. The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 2453 -- 2470 . ACL, 2020 . Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason We- ston. The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2453--2470. ACL, 2020.
- Congzheng Song and Ananth Raghunathan. Information Leakage in Embedding Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS) , pages 377 -- 390 . ACM, 2020 . Congzheng Song and Ananth Raghunathan. Information Leakage in Embedding Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 377--390. ACM, 2020.
- Hao Sun , Guangxuan Xu , Jiawen Deng , Jiale Cheng , Chujie Zheng , Hao Zhou , Nanyun Peng , Xiaoyan Zhu , and Minlie Huang . On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. CoRR abs/2110.08466 , 2021 . Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. CoRR abs/2110.08466, 2021. · arXiv
- Ilya Sutskever , Oriol Vinyals , and Quoc V. Le . Sequence to Sequence Learning with Neural Networks . In Annual Conference on Neural Information Processing Systems (NIPS) , pages 3104 -- 3112 . NIPS, 2014 . Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to Sequence Learning with Neural Networks. In Annual Conference on Neural Information Processing Systems (NIPS), pages 3104--3112. NIPS, 2014.
- 10.1145/3442381.3450024 · DOI
- Megan Ung , Jing Xu , and Y- Lan Boureau . SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures. CoRR abs/2110.07518 , 2021 . Megan Ung, Jing Xu, and Y-Lan Boureau. SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures. CoRR abs/2110.07518, 2021. · arXiv
- 10.18653/v1/D19-1062 · DOI · Paper page
- Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin . Attention is All you Need . In Annual Conference on Neural Information Processing Systems (NIPS) , pages 5998 -- 6008 . NIPS, 2017 . Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NIPS), pages 5998--6008. NIPS, 2017.
- 10.18653/v1/D19-1221 · DOI · Paper page
- Longyue Wang , Jinhua Du , Liangyou Li , Zhaopeng Tu , Andy Way , and Qun Liu. Semantics-Enhanced Task-Oriented Dialogue Translation: A Case Study on Hotel Booking. In International Joint Conference on Natural Language Processing (IJCNLP) , pages 33 -- 36 . ACL, 2017 . Longyue Wang, Jinhua Du, Liangyou Li, Zhaopeng Tu, Andy Way, and Qun Liu. Semantics-Enhanced Task-Oriented Dialogue Translation: A Case Study on Hotel Booking. In International Joint Conference on Natural Language Processing (IJCNLP), pages 33--36. ACL, 2017.
- Canwen Xu , Zexue He , Zhankui He , and Julian J . McAuley. Leashing the In- ner Demons: Self-Detoxification for Language Models . In AAAI Conference on Artificial Intelligence (AAAI) , pages 11530 -- 11537 . AAAI, 2022 . Canwen Xu, Zexue He, Zhankui He, and Julian J. McAuley. Leashing the In- ner Demons: Self-Detoxification for Language Models. In AAAI Conference on Artificial Intelligence (AAAI), pages 11530--11537. AAAI, 2022.
- Jing Xu , Da Ju , Margaret Li , Y- Lan Boureau , Jason Weston , and Emily Dinan . Recipes for Safety in Open-domain Chatbots. CoRR abs/2010.07079 , 2020 . Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for Safety in Open-domain Chatbots. CoRR abs/2010.07079, 2020. · arXiv
- 10.1609/aaai.v31i1.11182 · DOI · Paper page
- 10.1145/3278532.3278550 · DOI · Paper page
- Savvas Zannettou , Tristan Caulfield , Emiliano De Cristofaro , Nicolas Kourtellis , Ilias Leontiadis , Michael Sirivianos , Gianluca Stringhini , and Jeremy Blackburn. The Web Centipede: Understanding How Web Communities Influence Each Other Through the Lens of Mainstream and Alternative News Sources. In ACM Internet Measurement Conference (IMC) , pages 405 -- 417 . ACM, 2017 . Savvas Zannettou, Tristan Caulfield, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Michael Sirivianos, Gianluca Stringhini, and Jeremy Blackburn. The Web Centipede: Understanding How Web Communities Influence Each Other Through the Lens of Mainstream and Alternative News Sources. In ACM Internet Measurement Conference (IMC), pages 405--417. ACM, 2017.
- Savvas Zannettou , Mai ElSherief , Elizabeth M. Belding , Shirin Nilizadeh , and Gi- anluca Stringhini. Measuring and Characterizing Hate Speech on News Websites. In ACM Conference on Web Science (WebSci) , pages 125 -- 134 . ACM, 2020 . Savvas Zannettou, Mai ElSherief, Elizabeth M. Belding, Shirin Nilizadeh, and Gi- anluca Stringhini. Measuring and Characterizing Hate Speech on News Websites. In ACM Conference on Web Science (WebSci), pages 125--134. ACM, 2020.
- 10.1609/icwsm.v14i1.7343 · DOI · Paper page
- Yaoming Zhu , Sidi Lu , Lei Zheng , Jiaxian Guo , Weinan Zhang , Jun Wang , and Yong Yu. Texygen: A Benchmarking Platform for Text Generation Models. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , pages 1097 -- 1100 . ACM, 2018 . Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A Benchmarking Platform for Text Generation Models. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1097--1100. ACM, 2018.
- Caleb Ziems , Bing He , Sandeep Soni , and Srijan Kumar . Racism is a Virus: Anti- Asian Hate and Counterhate in Social Media during the COVID-19 Crisis. CoRR abs/2005.12423 , 2021 . Caleb Ziems, Bing He, Sandeep Soni, and Srijan Kumar. Racism is a Virus: Anti- Asian Hate and Counterhate in Social Media during the COVID-19 Crisis. CoRR abs/2005.12423, 2021. · arXiv
Cited by
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03825 · Cited by 861
Self-citationShen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
Self-citationMasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418
“Recent studies [23], [22], [48], [39], [36], [40] have drawn attention to the rise of ”jailbreak prompts,” ingeniously crafted to circumvent the restrictions placed on language models and coax them into performing tasks beyond their intended scope.”
BackgroundSelf-citationZhang, R., Li, H., Meng, H., Zhan, J., Gan, H., & Lee, Y.-C. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713429 · Cited by 123
“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”
BiasAsker: Measuring the Bias in Conversational AI System
Wan, Y., Wang, W., He, P., Gu, J., Bai, H., & Lyu, M. R. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616310 · Cited by 54
“Jovi and Breeno) do not provide application programming interface (API) access and can only be accessed manually.”
BackgroundMethodInfluential
Metadata
- Authors
- Si, W. M., Backes, M., Blackburn, J., De Cristofaro, E., Stringhini, G., Zannettou, S., & Zhang, Y.
- Venue
- Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2022-11-07
- Pages
- 2659-2673
- OpenAlex
- W4308391526
- Cited by
- 38
- References
- 24
- FWCI
- 5.09
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- NSF (National Science Foundation) (IIS-2046590 CNS-2114411 CNS-1942610 CNS-2114407)
- UK's National Research Centre on Privacy Harm Reduction and Adversarial Influence Online (EP/V011189/1)
- Helmholtz Association (ZT-I-OO1 4)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. https://app.cochat.ai/papers/10.1145/3548606.3560599
- The paper
- Si, W. M., Backes, M., Blackburn, J., De Cristofaro, E., Stringhini, G., Zannettou, S., & Zhang, Y. (2022). Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. https://doi.org/10.1145/3548606.3560599
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.