Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Chatbots are used in many applications, e.g., automated agents, smart home assistants, interactive characters in online games, etc. Therefore, it is crucial to ensure they do not behave in undesired manners, providing offensive or toxic responses to users. This is not a trivial task as state-of-the-art chatbot models are trained on large, public datasets openly collected from the Internet. This paper presents a first-of-its-kind, large-scale measurement of toxicity in chatbots. We show that publicly available chatbots are prone to providing toxic responses when fed toxic queries. Even more worryingly, some non-toxic queries can trigger toxic responses too. We then set out to design and experiment with an attack, ToxicBuddy, which relies on fine-tuning GPT-2 to generate non-toxic queries that make chatbots respond in a toxic manner. Our extensive experimental evaluation demonstrates that our attack is effective against public chatbot models and outperforms manually-crafted malicious queries proposed by previous work. We also evaluate three defense mechanisms against ToxicBuddy, showing that they either reduce the attack performance at the cost of affecting the chatbot's utility or are only effective at mitigating a portion of the attack. This highlights the need for more research from the computer security and online safety communities to ensure that chatbot models do not hurt their users. Overall, we are confident that ToxicBuddy can be used as an auditing tool and that our work will pave the way toward designing more effective defenses for chatbot safety.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 1×
“, vis-à-vis attacks like model extraction [61], membership inference attacks [55], and adversarial examples [41].”
BackgroundThe Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.30 · Cited by 1,076
Self-citationBaumgartner, J., Zannettou, S., Keegan, B., Squire, M., & Blackburn, J. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7347 · Cited by 881
Self-citationStealing Machine Learning Models via Prediction APIs
Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016) · arXiv · doi:10.48550/arxiv.1609.02943 · Cited by 796
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784
Recipes for building an open-domain chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Shuster, K., Smith, E. M., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.13637 · Cited by 784 · cited 14×
“Both BlenderBot and DialoGPT are pre-trained on a variant of the Reddit dataset.”
BackgroundMethodInfluentialMiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703 · cited 1×
“We cluster all the queries and embed each query by a popular pre-trained language understanding model all-MiniLML6-v2 [68].”
MethodRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Universal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642
A Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579
The Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 537 · cited 1×
“, by removing “contaminated” information from models [17, 69, 71] or prevent inappropriate text generation [20, 53].”
BackgroundRed Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363
Predictive biases in natural language processing models: A conceptual framework and overview
Shah, D. S., Schwartz, H. A., & Hovy, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.468 · Cited by 241 · cited 1×
“These tools are not perfect and, in some cases, biased [34, 52].”
BackgroundDiverse Beam Search for Improved Description of Complex Scenes
Vijayakumar, A., Cogswell, M., Selvaraju, R., Sun, Q., Lee, S., Crandall, D., & Batra, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12340 · Cited by 219 · cited 1×
“Finally, researchers have dedicated a lot of effort to studying diverse decoding methods for better response generation [24, 29, 65].”
BackgroundBuild it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
Dinan, E., Humeau, S., Chintagunta, B., & Weston, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1461 · Cited by 161 · cited 6×
“[16] use bias-controlled training to alleviate the problem of gender bias, while [17] develops a new training procedure to enhance chatbot models with crowd-workers iteratively.”
BackgroundMethodInfluentialChallenges in Detoxifying Language Models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., & Huang, P.-S. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.210 · Cited by 112
Bot-Adversarial Dialogue for Safe Conversational Agents
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.235 · Cited by 64
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2021) · arXiv · doi:10.48550/arxiv.2107.03451 · Cited by 44 · cited 7×
“, using safety classifiers assigning a score for queries and responses [15].”
BackgroundMethodInfluential
Research works printed but not linked: 52
- Yoshua Bengio , Réjean Ducharme , and Pascal Vincent. A Neural Probabilistic Language Model. In Annual Conference on Neural Information Processing Systems (NIPS) , pages 932 -- 938 . NIPS, 2000 . Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A Neural Probabilistic Language Model. In Annual Conference on Neural Information Processing Systems (NIPS), pages 932--938. NIPS, 2000.
- Antoine Bordes , Y- Lan Boureau , and Jason Weston. Learning End-to-End Goal- Oriented Dialog. In International Conference on Learning Representations (ICLR) , 2017 . Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learning End-to-End Goal- Oriented Dialog. In International Conference on Learning Representations (ICLR), 2017.
- Mohit Chandra , Dheeraj Reddy Pailla , Himanshu Bhatia , AadilMehdi J. Sanchawala , Manish Gupta , Manish Shrivastava , and Ponnurangam Kumaraguru . ? Subverting the Jewtocracy" : Online Antisemitism Detection Using Multimodal Deep Learning . In ACM Conference on Web Science (WebSci) , pages 148 -- 157 . ACM, 2021 . Mohit Chandra, Dheeraj Reddy Pailla, Himanshu Bhatia, AadilMehdi J. Sanchawala, Manish Gupta, Manish Shrivastava, and Ponnurangam Kumaraguru. ?Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learning. In ACM Conference on Web Science (WebSci), pages 148--157. ACM, 2021.
- 10.1145/3465336.3475111 · DOI
- Despoina Chatzakou , Ilias Leontiadis , Jeremy Blackburn , Emiliano De Cristofaro , Gianluca Stringhini, Athena Vakali, and Nicolas Kourtellis. Detecting Cyberbullying and Cyberaggression in Social Media. CoRR abs/1907.08873 , 2019 . Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Athena Vakali, and Nicolas Kourtellis. Detecting Cyberbullying and Cyberaggression in Social Media. CoRR abs/1907.08873, 2019. · arXiv
- 10.18653/v1/2020.emnlp-main.656 · DOI · Paper page
- Markus Freitag and Yaser Al-Onaizan . Beam Search Strategies for Neural Machine Translation. CoRR abs/1702.01806 , 2017 . Markus Freitag and Yaser Al-Onaizan. Beam Search Strategies for Neural Machine Translation. CoRR abs/1702.01806, 2017. · arXiv
- Jianfeng Gao , Michel Galley , and Lihong Li . Neural Approaches to Conversational AI . In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 2 -- 7 . ACL, 2018 . Jianfeng Gao, Michel Galley, and Lihong Li. Neural Approaches to Conversational AI. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2--7. ACL, 2018.
- Tianxing He and James R. Glass. Detecting Egregious Responses in Neural Sequence-to-sequence Models. In International Conference on Learning Representations (ICLR) , 2019 . Tianxing He and James R. Glass. Detecting Egregious Responses in Neural Sequence-to-sequence Models. In International Conference on Learning Representations (ICLR), 2019.
- Gabriel Emile Hine , Jeremiah Onaolapo , Emiliano De Cristofaro , Nicolas Kourtel- lis, Ilias Leontiadis , Riginos Samaras , Gianluca Stringhini , and Jeremy Blackburn . Kek, Cucks, and God Emperor Trump : A Measurement Study of 4chan's Politi- cally Incorrect Forum and Its Effects on the Web . In International Conference on Web and Social Media (ICWSM) , pages 92 -- 101 . AAAI, 2017 . Gabriel Emile Hine, Jeremiah Onaolapo, Emiliano De Cristofaro, Nicolas Kourtel- lis, Ilias Leontiadis, Riginos Samaras, Gianluca Stringhini, and Jeremy Blackburn. Kek, Cucks, and God Emperor Trump: A Measurement Study of 4chan's Politi- cally Incorrect Forum and Its Effects on the Web. In International Conference on Web and Social Media (ICWSM), pages 92--101. AAAI, 2017.
- Geoffrey E. Hinton , Oriol Vinyals , and Jeffrey Dean . Distilling the Knowledge in a Neural Network. CoRR abs/1503.02531 , 2015 . Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. CoRR abs/1503.02531, 2015. · arXiv
- Hossein Hosseini , Sreeram Kannan , Baosen Zhang , and Radha Poovendran . Deceiving Google's Perspective API Built for Detecting Toxic Comments. CoRR abs/1702.08138 , 2017 . Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. Deceiving Google's Perspective API Built for Detecting Toxic Comments. CoRR abs/1702.08138, 2017. · arXiv
- Chandra Khatri , Behnam Hedayatnia , Anu Venkatesh , Jeff Nunn , Yi Pan , Qing Liu , Han Song , Anna Gottardi , Sanjeev Kwatra , Sanju Pancholi , Ming Cheng , Qinglang Chen , Lauren Stubel , Karthik Gopalakrishnan , Kate Bland , Raefer Gabriel , Arindam Mandal , Dilek Hakkani-Tür , Gene Hwang , Nate Michel , Eric King , and Rohit Prasad . Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize. CoRR abs/1812.10757 , 2018 . Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, Ming Cheng, Qinglang Chen, Lauren Stubel, Karthik Gopalakrishnan, Kate Bland, Raefer Gabriel, Arindam Mandal, Dilek Hakkani-Tür, Gene Hwang, Nate Michel, Eric King, and Rohit Prasad. Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize. CoRR abs/1812.10757, 2018. · arXiv
- Virapat Kieuvongngam , Bowen Tan , and Yiming Niu . Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2. CoRR abs/2006.01997 , 2020 . Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2. CoRR abs/2006.01997, 2020. · arXiv
- 10.18653/v1/N16-1014 · DOI · Paper page
- Jiwei Li , Will Monroe , and Dan Jurafsky . A Simple , Fast Diverse Decoding Algorithm for Neural Generation. CoRR abs/1611.08562 , 2016 . Jiwei Li, Will Monroe, and Dan Jurafsky. A Simple, Fast Diverse Decoding Algorithm for Neural Generation. CoRR abs/1611.08562, 2016. · arXiv
- Jiwei Li , Will Monroe , Alan Ritter , Dan Jurafsky , Michel Galley , and Jianfeng Gao. Deep Reinforcement Learning for Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1192 -- 1202 . ACL, 2016 . Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep Reinforcement Learning for Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1192--1202. ACL, 2016.
- Haochen Liu , Tyler Derr , Zitao Liu , and Jiliang Tang . Say What I Want: Towards the Dark Side of Neural Dialogue Models. CoRR abs/1909.06044 , 2019 . Haochen Liu, Tyler Derr, Zitao Liu, and Jiliang Tang. Say What I Want: Towards the Dark Side of Neural Dialogue Models. CoRR abs/1909.06044, 2019. · arXiv
- 10.18653/v1/2022.naacl-main.204 · DOI
- Alexander H. Miller , Will Feng , Adam Fisch , Jiasen Lu , Dhruv Batra , Antoine Bordes , Devi Parikh , and Jason Weston . ParlAI: A Dialog Research Software Platform. CoRR abs/1705.06476 , 2017 . Alexander H. Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. ParlAI: A Dialog Research Software Platform. CoRR abs/1705.06476, 2017. · arXiv
- Alexandros Mittos , Savvas Zannettou , Jeremy Blackburn , and Emiliano De Cristo- faro. " And We Will Fight For Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In International Conference on Web and Social Media (ICWSM) , pages 452 -- 463 . AAAI, 2020 . Alexandros Mittos, Savvas Zannettou, Jeremy Blackburn, and Emiliano De Cristo- faro. "And We Will Fight For Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In International Conference on Web and Social Media (ICWSM), pages 452--463. AAAI, 2020.
- Abby Ohlheiser . Trolls turned Tay , Microsoft's fun millennial AI bot, into a genocidal maniac. https://wapo.st/3mRIOww , 2016 . Abby Ohlheiser. Trolls turned Tay, Microsoft's fun millennial AI bot, into a genocidal maniac. https://wapo.st/3mRIOww, 2016.
- Nedjma Ousidhoum , Xinran Zhao , Tianqing Fang , Yangqiu Song , and Dit-Yan Yeung. Probing Toxic Content in Large Pre-Trained Language Models. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 4262 -- 4274 . ACL, 2021 . Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing Toxic Content in Large Pre-Trained Language Models. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4262--4274. ACL, 2021.
- 10.1609/icwsm.v16i1.19330 · DOI
- 10.1609/icwsm.v14i1.7354 · DOI
- 10.1109/SP.2016.41 · DOI · Paper page
- Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . Bleu : a Method for Automatic Evaluation of Machine Translation . In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 311 -- 318 . ACL, 2016 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a Method for Automatic Evaluation of Machine Translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 311--318. ACL, 2016.
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners. OpenAI blog , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI blog, 2019.
- Ashwin Rajadesingan , Paul Resnick , and Ceren Budak . Quick , Community-Specific Learning: How Distinctive Toxicity Norms Are Maintained in Political Subreddits. In International Conference on Web and Social Media (ICWSM) , pages 557 -- 568 . AAAI, 2020 . Ashwin Rajadesingan, Paul Resnick, and Ceren Budak. Quick, Community-Specific Learning: How Distinctive Toxicity Norms Are Maintained in Political Subreddits. In International Conference on Web and Social Media (ICWSM), pages 557--568. AAAI, 2020.
- Manoel Horta Ribeiro , Jeremy Blackburn , Barry Bradlyn , Emiliano De Cristofaro , Gianluca Stringhini , Summer Long , Stephanie Greenberg , and Savvas Zannettou . The Evolution of the Manosphere across the Web . In International Conference on Web and Social Media (ICWSM) , pages 196 -- 207 . AAAI, 2021 . Manoel Horta Ribeiro, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro, Gianluca Stringhini, Summer Long, Stephanie Greenberg, and Savvas Zannettou. The Evolution of the Manosphere across the Web. In International Conference on Web and Social Media (ICWSM), pages 196--207. AAAI, 2021.
- Alan Ritter , Colin Cherry , and Bill Dolan. Unsupervised Modeling of Twitter Conversations. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages 172 -- 180 . ACL, 2010 . Alan Ritter, Colin Cherry, and Bill Dolan. Unsupervised Modeling of Twitter Conversations. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 172--180. ACL, 2010.
- Sascha Rothe , Shashi Narayan , and Aliaksei Severyn . Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. CoRR abs/1907.12461 , 2019 . Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. CoRR abs/1907.12461, 2019. · arXiv
- Maarten Sap , Dallas Card , Saadia Gabriel , Yejin Choi , and Noah A . Smith. The Risk of Racial Bias in Hate Speech Detection. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 1668 -- 1678 . ACL, 2019 . Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. The Risk of Racial Bias in Hate Speech Detection. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1668--1678. ACL, 2019.
- Emily Sheng , Kai-Wei Chang , Premkumar Natarajan , and Nanyun Peng . Towards Controllable Biases in Language Generation. CoRR abs/2005.00268 , 2020 . Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. Towards Controllable Biases in Language Generation. CoRR abs/2005.00268, 2020. · arXiv
- Kurt Shuster , Da Ju , Stephen Roller , Emily Dinan , Y- Lan Boureau , and Jason We- ston. The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 2453 -- 2470 . ACL, 2020 . Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason We- ston. The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2453--2470. ACL, 2020.
- Congzheng Song and Ananth Raghunathan. Information Leakage in Embedding Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS) , pages 377 -- 390 . ACM, 2020 . Congzheng Song and Ananth Raghunathan. Information Leakage in Embedding Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 377--390. ACM, 2020.
- Hao Sun , Guangxuan Xu , Jiawen Deng , Jiale Cheng , Chujie Zheng , Hao Zhou , Nanyun Peng , Xiaoyan Zhu , and Minlie Huang . On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. CoRR abs/2110.08466 , 2021 . Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. CoRR abs/2110.08466, 2021. · arXiv
- Ilya Sutskever , Oriol Vinyals , and Quoc V. Le . Sequence to Sequence Learning with Neural Networks . In Annual Conference on Neural Information Processing Systems (NIPS) , pages 3104 -- 3112 . NIPS, 2014 . Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to Sequence Learning with Neural Networks. In Annual Conference on Neural Information Processing Systems (NIPS), pages 3104--3112. NIPS, 2014.
- 10.1145/3442381.3450024 · DOI
- Megan Ung , Jing Xu , and Y- Lan Boureau . SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures. CoRR abs/2110.07518 , 2021 . Megan Ung, Jing Xu, and Y-Lan Boureau. SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures. CoRR abs/2110.07518, 2021. · arXiv
- 10.18653/v1/D19-1062 · DOI · Paper page
- Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin . Attention is All you Need . In Annual Conference on Neural Information Processing Systems (NIPS) , pages 5998 -- 6008 . NIPS, 2017 . Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NIPS), pages 5998--6008. NIPS, 2017.
- Longyue Wang , Jinhua Du , Liangyou Li , Zhaopeng Tu , Andy Way , and Qun Liu. Semantics-Enhanced Task-Oriented Dialogue Translation: A Case Study on Hotel Booking. In International Joint Conference on Natural Language Processing (IJCNLP) , pages 33 -- 36 . ACL, 2017 . Longyue Wang, Jinhua Du, Liangyou Li, Zhaopeng Tu, Andy Way, and Qun Liu. Semantics-Enhanced Task-Oriented Dialogue Translation: A Case Study on Hotel Booking. In International Joint Conference on Natural Language Processing (IJCNLP), pages 33--36. ACL, 2017.
- Canwen Xu , Zexue He , Zhankui He , and Julian J . McAuley. Leashing the In- ner Demons: Self-Detoxification for Language Models . In AAAI Conference on Artificial Intelligence (AAAI) , pages 11530 -- 11537 . AAAI, 2022 . Canwen Xu, Zexue He, Zhankui He, and Julian J. McAuley. Leashing the In- ner Demons: Self-Detoxification for Language Models. In AAAI Conference on Artificial Intelligence (AAAI), pages 11530--11537. AAAI, 2022.
- Jing Xu , Da Ju , Margaret Li , Y- Lan Boureau , Jason Weston , and Emily Dinan . Recipes for Safety in Open-domain Chatbots. CoRR abs/2010.07079 , 2020 . Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for Safety in Open-domain Chatbots. CoRR abs/2010.07079, 2020. · arXiv
- 10.1609/aaai.v31i1.11182 · DOI · Paper page
- 10.1145/3278532.3278550 · DOI · Paper page
- Savvas Zannettou , Tristan Caulfield , Emiliano De Cristofaro , Nicolas Kourtellis , Ilias Leontiadis , Michael Sirivianos , Gianluca Stringhini , and Jeremy Blackburn. The Web Centipede: Understanding How Web Communities Influence Each Other Through the Lens of Mainstream and Alternative News Sources. In ACM Internet Measurement Conference (IMC) , pages 405 -- 417 . ACM, 2017 . Savvas Zannettou, Tristan Caulfield, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Michael Sirivianos, Gianluca Stringhini, and Jeremy Blackburn. The Web Centipede: Understanding How Web Communities Influence Each Other Through the Lens of Mainstream and Alternative News Sources. In ACM Internet Measurement Conference (IMC), pages 405--417. ACM, 2017.
- Savvas Zannettou , Mai ElSherief , Elizabeth M. Belding , Shirin Nilizadeh , and Gi- anluca Stringhini. Measuring and Characterizing Hate Speech on News Websites. In ACM Conference on Web Science (WebSci) , pages 125 -- 134 . ACM, 2020 . Savvas Zannettou, Mai ElSherief, Elizabeth M. Belding, Shirin Nilizadeh, and Gi- anluca Stringhini. Measuring and Characterizing Hate Speech on News Websites. In ACM Conference on Web Science (WebSci), pages 125--134. ACM, 2020.
- 10.1609/icwsm.v14i1.7343 · DOI · Paper page
- Yaoming Zhu , Sidi Lu , Lei Zheng , Jiaxian Guo , Weinan Zhang , Jun Wang , and Yong Yu. Texygen: A Benchmarking Platform for Text Generation Models. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , pages 1097 -- 1100 . ACM, 2018 . Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A Benchmarking Platform for Text Generation Models. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1097--1100. ACM, 2018.
- Caleb Ziems , Bing He , Sandeep Soni , and Srijan Kumar . Racism is a Virus: Anti- Asian Hate and Counterhate in Social Media during the COVID-19 Crisis. CoRR abs/2005.12423 , 2021 . Caleb Ziems, Bing He, Sandeep Soni, and Srijan Kumar. Racism is a Virus: Anti- Asian Hate and Counterhate in Social Media during the COVID-19 Crisis. CoRR abs/2005.12423, 2021. · arXiv
Cited by
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03825 · Cited by 861
Self-citationShen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
Self-citationMasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots
Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., & Liu, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.08715 · Cited by 418
“Recent studies [23], [22], [48], [39], [36], [40] have drawn attention to the rise of ”jailbreak prompts,” ingeniously crafted to circumvent the restrictions placed on language models and coax them into performing tasks beyond their intended scope.”
BackgroundSelf-citationZhang, R., Li, H., Meng, H., Zhan, J., Gan, H., & Lee, Y.-C. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713429 · Cited by 123
“To address this, we propose a role-based approach to studying AI companion harms, which is crucial for identifying the root causes of harm and AI’s responsibility in generating harm.”
BiasAsker: Measuring the Bias in Conversational AI System
Wan, Y., Wang, W., He, P., Gu, J., Bai, H., & Lyu, M. R. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616310 · Cited by 54
“Jovi and Breeno) do not provide application programming interface (API) access and can only be accessed manually.”
BackgroundMethodInfluential
Metadata
- Authors
- Si, W. M., Backes, M., Blackburn, J., De Cristofaro, E., Stringhini, G., Zannettou, S., & Zhang, Y.
- Venue
- Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2022-11-07
- Pages
- 2659-2673
- OpenAlex
- W4308391526
- Cited by
- 38
- References
- 25
- FWCI
- 5.09
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- NSF (National Science Foundation) (IIS-2046590 CNS-2114411 CNS-1942610 CNS-2114407)
- UK's National Research Centre on Privacy Harm Reduction and Adversarial Influence Online (EP/V011189/1)
- Helmholtz Association (ZT-I-OO1 4)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.