CoChat

Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots

Open access
Si, Wai Man, Backes, Michael, Blackburn, Jeremy, De Cristofaro, Emiliano, Stringhini, Gianluca, Zannettou, Savvas, Zhang, Yang · 2022 · Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security
Cited by 38

The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.

Abstract

Chatbots are used in many applications, e.g., automated agents, smart home assistants, interactive characters in online games, etc. Therefore, it is crucial to ensure they do not behave in undesired manners, providing offensive or toxic responses to users. This is not a trivial task as state-of-the-art chatbot models are trained on large, public datasets openly collected from the Internet. This paper presents a first-of-its-kind, large-scale measurement of toxicity in chatbots. We show that publicly available chatbots are prone to providing toxic responses when fed toxic queries. Even more worryingly, some non-toxic queries can trigger toxic responses too. We then set out to design and experiment with an attack, ToxicBuddy, which relies on fine-tuning GPT-2 to generate non-toxic queries that make chatbots respond in a toxic manner. Our extensive experimental evaluation demonstrates that our attack is effective against public chatbot models and outperforms manually-crafted malicious queries proposed by previous work. We also evaluate three defense mechanisms against ToxicBuddy, showing that they either reduce the attack performance at the cost of affecting the chatbot's utility or are only effective at mitigating a portion of the attack. This highlights the need for more research from the computer security and online safety communities to ensure that chatbot models do not hurt their users. Overall, we are confident that ToxicBuddy can be used as an auditing tool and that our work will pave the way toward designing more effective defenses for chatbot safety.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3548606.3560599

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 52

  1. Yoshua Bengio , Réjean Ducharme , and Pascal Vincent. A Neural Probabilistic Language Model. In Annual Conference on Neural Information Processing Systems (NIPS) , pages 932 -- 938 . NIPS, 2000 . Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A Neural Probabilistic Language Model. In Annual Conference on Neural Information Processing Systems (NIPS), pages 932--938. NIPS, 2000.
  2. Antoine Bordes , Y- Lan Boureau , and Jason Weston. Learning End-to-End Goal- Oriented Dialog. In International Conference on Learning Representations (ICLR) , 2017 . Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learning End-to-End Goal- Oriented Dialog. In International Conference on Learning Representations (ICLR), 2017.
  3. Mohit Chandra , Dheeraj Reddy Pailla , Himanshu Bhatia , AadilMehdi J. Sanchawala , Manish Gupta , Manish Shrivastava , and Ponnurangam Kumaraguru . ? Subverting the Jewtocracy" : Online Antisemitism Detection Using Multimodal Deep Learning . In ACM Conference on Web Science (WebSci) , pages 148 -- 157 . ACM, 2021 . Mohit Chandra, Dheeraj Reddy Pailla, Himanshu Bhatia, AadilMehdi J. Sanchawala, Manish Gupta, Manish Shrivastava, and Ponnurangam Kumaraguru. ?Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learning. In ACM Conference on Web Science (WebSci), pages 148--157. ACM, 2021.
  4. 10.1145/3465336.3475111 · DOI
  5. Despoina Chatzakou , Ilias Leontiadis , Jeremy Blackburn , Emiliano De Cristofaro , Gianluca Stringhini, Athena Vakali, and Nicolas Kourtellis. Detecting Cyberbullying and Cyberaggression in Social Media. CoRR abs/1907.08873 , 2019 . Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Athena Vakali, and Nicolas Kourtellis. Detecting Cyberbullying and Cyberaggression in Social Media. CoRR abs/1907.08873, 2019. · arXiv
  6. 10.18653/v1/2020.emnlp-main.656 · DOI · Paper page
  7. Markus Freitag and Yaser Al-Onaizan . Beam Search Strategies for Neural Machine Translation. CoRR abs/1702.01806 , 2017 . Markus Freitag and Yaser Al-Onaizan. Beam Search Strategies for Neural Machine Translation. CoRR abs/1702.01806, 2017. · arXiv
  8. Jianfeng Gao , Michel Galley , and Lihong Li . Neural Approaches to Conversational AI . In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 2 -- 7 . ACL, 2018 . Jianfeng Gao, Michel Galley, and Lihong Li. Neural Approaches to Conversational AI. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2--7. ACL, 2018.
  9. Tianxing He and James R. Glass. Detecting Egregious Responses in Neural Sequence-to-sequence Models. In International Conference on Learning Representations (ICLR) , 2019 . Tianxing He and James R. Glass. Detecting Egregious Responses in Neural Sequence-to-sequence Models. In International Conference on Learning Representations (ICLR), 2019.
  10. Gabriel Emile Hine , Jeremiah Onaolapo , Emiliano De Cristofaro , Nicolas Kourtel- lis, Ilias Leontiadis , Riginos Samaras , Gianluca Stringhini , and Jeremy Blackburn . Kek, Cucks, and God Emperor Trump : A Measurement Study of 4chan's Politi- cally Incorrect Forum and Its Effects on the Web . In International Conference on Web and Social Media (ICWSM) , pages 92 -- 101 . AAAI, 2017 . Gabriel Emile Hine, Jeremiah Onaolapo, Emiliano De Cristofaro, Nicolas Kourtel- lis, Ilias Leontiadis, Riginos Samaras, Gianluca Stringhini, and Jeremy Blackburn. Kek, Cucks, and God Emperor Trump: A Measurement Study of 4chan's Politi- cally Incorrect Forum and Its Effects on the Web. In International Conference on Web and Social Media (ICWSM), pages 92--101. AAAI, 2017.
  11. Geoffrey E. Hinton , Oriol Vinyals , and Jeffrey Dean . Distilling the Knowledge in a Neural Network. CoRR abs/1503.02531 , 2015 . Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. CoRR abs/1503.02531, 2015. · arXiv
  12. Hossein Hosseini , Sreeram Kannan , Baosen Zhang , and Radha Poovendran . Deceiving Google's Perspective API Built for Detecting Toxic Comments. CoRR abs/1702.08138 , 2017 . Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. Deceiving Google's Perspective API Built for Detecting Toxic Comments. CoRR abs/1702.08138, 2017. · arXiv
  13. Chandra Khatri , Behnam Hedayatnia , Anu Venkatesh , Jeff Nunn , Yi Pan , Qing Liu , Han Song , Anna Gottardi , Sanjeev Kwatra , Sanju Pancholi , Ming Cheng , Qinglang Chen , Lauren Stubel , Karthik Gopalakrishnan , Kate Bland , Raefer Gabriel , Arindam Mandal , Dilek Hakkani-Tür , Gene Hwang , Nate Michel , Eric King , and Rohit Prasad . Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize. CoRR abs/1812.10757 , 2018 . Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, Ming Cheng, Qinglang Chen, Lauren Stubel, Karthik Gopalakrishnan, Kate Bland, Raefer Gabriel, Arindam Mandal, Dilek Hakkani-Tür, Gene Hwang, Nate Michel, Eric King, and Rohit Prasad. Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize. CoRR abs/1812.10757, 2018. · arXiv
  14. Virapat Kieuvongngam , Bowen Tan , and Yiming Niu . Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2. CoRR abs/2006.01997 , 2020 . Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2. CoRR abs/2006.01997, 2020. · arXiv
  15. 10.18653/v1/N16-1014 · DOI · Paper page
  16. Jiwei Li , Will Monroe , and Dan Jurafsky . A Simple , Fast Diverse Decoding Algorithm for Neural Generation. CoRR abs/1611.08562 , 2016 . Jiwei Li, Will Monroe, and Dan Jurafsky. A Simple, Fast Diverse Decoding Algorithm for Neural Generation. CoRR abs/1611.08562, 2016. · arXiv
  17. Jiwei Li , Will Monroe , Alan Ritter , Dan Jurafsky , Michel Galley , and Jianfeng Gao. Deep Reinforcement Learning for Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1192 -- 1202 . ACL, 2016 . Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep Reinforcement Learning for Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1192--1202. ACL, 2016.
  18. Haochen Liu , Tyler Derr , Zitao Liu , and Jiliang Tang . Say What I Want: Towards the Dark Side of Neural Dialogue Models. CoRR abs/1909.06044 , 2019 . Haochen Liu, Tyler Derr, Zitao Liu, and Jiliang Tang. Say What I Want: Towards the Dark Side of Neural Dialogue Models. CoRR abs/1909.06044, 2019. · arXiv
  19. 10.18653/v1/2022.naacl-main.204 · DOI
  20. Alexander H. Miller , Will Feng , Adam Fisch , Jiasen Lu , Dhruv Batra , Antoine Bordes , Devi Parikh , and Jason Weston . ParlAI: A Dialog Research Software Platform. CoRR abs/1705.06476 , 2017 . Alexander H. Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. ParlAI: A Dialog Research Software Platform. CoRR abs/1705.06476, 2017. · arXiv
  21. Alexandros Mittos , Savvas Zannettou , Jeremy Blackburn , and Emiliano De Cristo- faro. " And We Will Fight For Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In International Conference on Web and Social Media (ICWSM) , pages 452 -- 463 . AAAI, 2020 . Alexandros Mittos, Savvas Zannettou, Jeremy Blackburn, and Emiliano De Cristo- faro. "And We Will Fight For Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In International Conference on Web and Social Media (ICWSM), pages 452--463. AAAI, 2020.
  22. Abby Ohlheiser . Trolls turned Tay , Microsoft's fun millennial AI bot, into a genocidal maniac. https://wapo.st/3mRIOww , 2016 . Abby Ohlheiser. Trolls turned Tay, Microsoft's fun millennial AI bot, into a genocidal maniac. https://wapo.st/3mRIOww, 2016.
  23. Nedjma Ousidhoum , Xinran Zhao , Tianqing Fang , Yangqiu Song , and Dit-Yan Yeung. Probing Toxic Content in Large Pre-Trained Language Models. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 4262 -- 4274 . ACL, 2021 . Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing Toxic Content in Large Pre-Trained Language Models. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4262--4274. ACL, 2021.
  24. 10.1609/icwsm.v16i1.19330 · DOI
  25. 10.1609/icwsm.v14i1.7354 · DOI
  26. 10.1109/SP.2016.41 · DOI · Paper page
  27. Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . Bleu : a Method for Automatic Evaluation of Machine Translation . In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 311 -- 318 . ACL, 2016 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a Method for Automatic Evaluation of Machine Translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 311--318. ACL, 2016.
  28. Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever . Language Models are Unsupervised Multitask Learners. OpenAI blog , 2019 . Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI blog, 2019.
  29. Ashwin Rajadesingan , Paul Resnick , and Ceren Budak . Quick , Community-Specific Learning: How Distinctive Toxicity Norms Are Maintained in Political Subreddits. In International Conference on Web and Social Media (ICWSM) , pages 557 -- 568 . AAAI, 2020 . Ashwin Rajadesingan, Paul Resnick, and Ceren Budak. Quick, Community-Specific Learning: How Distinctive Toxicity Norms Are Maintained in Political Subreddits. In International Conference on Web and Social Media (ICWSM), pages 557--568. AAAI, 2020.
  30. Manoel Horta Ribeiro , Jeremy Blackburn , Barry Bradlyn , Emiliano De Cristofaro , Gianluca Stringhini , Summer Long , Stephanie Greenberg , and Savvas Zannettou . The Evolution of the Manosphere across the Web . In International Conference on Web and Social Media (ICWSM) , pages 196 -- 207 . AAAI, 2021 . Manoel Horta Ribeiro, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro, Gianluca Stringhini, Summer Long, Stephanie Greenberg, and Savvas Zannettou. The Evolution of the Manosphere across the Web. In International Conference on Web and Social Media (ICWSM), pages 196--207. AAAI, 2021.
  31. Alan Ritter , Colin Cherry , and Bill Dolan. Unsupervised Modeling of Twitter Conversations. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages 172 -- 180 . ACL, 2010 . Alan Ritter, Colin Cherry, and Bill Dolan. Unsupervised Modeling of Twitter Conversations. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 172--180. ACL, 2010.
  32. Sascha Rothe , Shashi Narayan , and Aliaksei Severyn . Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. CoRR abs/1907.12461 , 2019 . Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. CoRR abs/1907.12461, 2019. · arXiv
  33. Maarten Sap , Dallas Card , Saadia Gabriel , Yejin Choi , and Noah A . Smith. The Risk of Racial Bias in Hate Speech Detection. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 1668 -- 1678 . ACL, 2019 . Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. The Risk of Racial Bias in Hate Speech Detection. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1668--1678. ACL, 2019.
  34. Emily Sheng , Kai-Wei Chang , Premkumar Natarajan , and Nanyun Peng . Towards Controllable Biases in Language Generation. CoRR abs/2005.00268 , 2020 . Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. Towards Controllable Biases in Language Generation. CoRR abs/2005.00268, 2020. · arXiv
  35. Kurt Shuster , Da Ju , Stephen Roller , Emily Dinan , Y- Lan Boureau , and Jason We- ston. The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 2453 -- 2470 . ACL, 2020 . Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason We- ston. The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2453--2470. ACL, 2020.
  36. Congzheng Song and Ananth Raghunathan. Information Leakage in Embedding Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS) , pages 377 -- 390 . ACM, 2020 . Congzheng Song and Ananth Raghunathan. Information Leakage in Embedding Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 377--390. ACM, 2020.
  37. Hao Sun , Guangxuan Xu , Jiawen Deng , Jiale Cheng , Chujie Zheng , Hao Zhou , Nanyun Peng , Xiaoyan Zhu , and Minlie Huang . On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. CoRR abs/2110.08466 , 2021 . Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. CoRR abs/2110.08466, 2021. · arXiv
  38. Ilya Sutskever , Oriol Vinyals , and Quoc V. Le . Sequence to Sequence Learning with Neural Networks . In Annual Conference on Neural Information Processing Systems (NIPS) , pages 3104 -- 3112 . NIPS, 2014 . Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to Sequence Learning with Neural Networks. In Annual Conference on Neural Information Processing Systems (NIPS), pages 3104--3112. NIPS, 2014.
  39. 10.1145/3442381.3450024 · DOI
  40. Megan Ung , Jing Xu , and Y- Lan Boureau . SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures. CoRR abs/2110.07518 , 2021 . Megan Ung, Jing Xu, and Y-Lan Boureau. SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures. CoRR abs/2110.07518, 2021. · arXiv
  41. 10.18653/v1/D19-1062 · DOI · Paper page
  42. Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin . Attention is All you Need . In Annual Conference on Neural Information Processing Systems (NIPS) , pages 5998 -- 6008 . NIPS, 2017 . Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NIPS), pages 5998--6008. NIPS, 2017.
  43. Longyue Wang , Jinhua Du , Liangyou Li , Zhaopeng Tu , Andy Way , and Qun Liu. Semantics-Enhanced Task-Oriented Dialogue Translation: A Case Study on Hotel Booking. In International Joint Conference on Natural Language Processing (IJCNLP) , pages 33 -- 36 . ACL, 2017 . Longyue Wang, Jinhua Du, Liangyou Li, Zhaopeng Tu, Andy Way, and Qun Liu. Semantics-Enhanced Task-Oriented Dialogue Translation: A Case Study on Hotel Booking. In International Joint Conference on Natural Language Processing (IJCNLP), pages 33--36. ACL, 2017.
  44. Canwen Xu , Zexue He , Zhankui He , and Julian J . McAuley. Leashing the In- ner Demons: Self-Detoxification for Language Models . In AAAI Conference on Artificial Intelligence (AAAI) , pages 11530 -- 11537 . AAAI, 2022 . Canwen Xu, Zexue He, Zhankui He, and Julian J. McAuley. Leashing the In- ner Demons: Self-Detoxification for Language Models. In AAAI Conference on Artificial Intelligence (AAAI), pages 11530--11537. AAAI, 2022.
  45. Jing Xu , Da Ju , Margaret Li , Y- Lan Boureau , Jason Weston , and Emily Dinan . Recipes for Safety in Open-domain Chatbots. CoRR abs/2010.07079 , 2020 . Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for Safety in Open-domain Chatbots. CoRR abs/2010.07079, 2020. · arXiv
  46. 10.1609/aaai.v31i1.11182 · DOI · Paper page
  47. 10.1145/3278532.3278550 · DOI · Paper page
  48. Savvas Zannettou , Tristan Caulfield , Emiliano De Cristofaro , Nicolas Kourtellis , Ilias Leontiadis , Michael Sirivianos , Gianluca Stringhini , and Jeremy Blackburn. The Web Centipede: Understanding How Web Communities Influence Each Other Through the Lens of Mainstream and Alternative News Sources. In ACM Internet Measurement Conference (IMC) , pages 405 -- 417 . ACM, 2017 . Savvas Zannettou, Tristan Caulfield, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Michael Sirivianos, Gianluca Stringhini, and Jeremy Blackburn. The Web Centipede: Understanding How Web Communities Influence Each Other Through the Lens of Mainstream and Alternative News Sources. In ACM Internet Measurement Conference (IMC), pages 405--417. ACM, 2017.
  49. Savvas Zannettou , Mai ElSherief , Elizabeth M. Belding , Shirin Nilizadeh , and Gi- anluca Stringhini. Measuring and Characterizing Hate Speech on News Websites. In ACM Conference on Web Science (WebSci) , pages 125 -- 134 . ACM, 2020 . Savvas Zannettou, Mai ElSherief, Elizabeth M. Belding, Shirin Nilizadeh, and Gi- anluca Stringhini. Measuring and Characterizing Hate Speech on News Websites. In ACM Conference on Web Science (WebSci), pages 125--134. ACM, 2020.
  50. 10.1609/icwsm.v14i1.7343 · DOI · Paper page
  51. Yaoming Zhu , Sidi Lu , Lei Zheng , Jiaxian Guo , Weinan Zhang , Jun Wang , and Yong Yu. Texygen: A Benchmarking Platform for Text Generation Models. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , pages 1097 -- 1100 . ACM, 2018 . Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A Benchmarking Platform for Text Generation Models. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1097--1100. ACM, 2018.
  52. Caleb Ziems , Bing He , Sandeep Soni , and Srijan Kumar . Racism is a Virus: Anti- Asian Hate and Counterhate in Social Media during the COVID-19 Crisis. CoRR abs/2005.12423 , 2021 . Caleb Ziems, Bing He, Sandeep Soni, and Srijan Kumar. Racism is a Virus: Anti- Asian Hate and Counterhate in Social Media during the COVID-19 Crisis. CoRR abs/2005.12423, 2021. · arXiv

Cited by

See all 38

Metadata

Authors
Si, W. M., Backes, M., Blackburn, J., De Cristofaro, E., Stringhini, G., Zannettou, S., & Zhang, Y.
Venue
Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security
Publisher
ACM
Published
2022-11-07
Pages
2659-2673
OpenAlex
W4308391526
Cited by
38
References
25
FWCI
5.09
Type
conference paper
Language
en
Open access
green
Funders
  • NSF (National Science Foundation) (IIS-2046590 CNS-2114411 CNS-1942610 CNS-2114407)
  • UK's National Research Centre on Privacy Harm Reduction and Adversarial Influence Online (EP/V011189/1)
  • Helmholtz Association (ZT-I-OO1 4)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.