ProsocialDialog: A prosocial backbone for conversational agents
Open accessAbstract
Most existing dialogue systems fail to respond properly to potentially unsafe user utterances by either ignoring or passively agreeing with them.To address this issue, we introduce PROSOCIALDIALOG, the first large-scale multi-turn dialogue dataset to teach conversational agents to respond to problematic content following social norms.Covering diverse unethical, problematic, biased, and toxic situations, PROSOCIALDIALOG contains responses that encourage prosocial behavior, grounded in commonsense social rules (i.e., rules-ofthumb, RoTs).Created via a human-AI collaborative framework, PROSOCIALDIALOG consists of 58K dialogues, with 331K utterances, 160K unique RoTs, and 497K dialogue safety labels accompanied by free-form rationales.With this dataset, we introduce a dialogue safety detection module, Canary, capable of generating RoTs given conversational context, and a socially-informed dialogue agent, Prost.Empirical results show that Prost generates more socially acceptable dialogues compared to other state-of-the-art language and dialogue models in both in-domain and out-of-domain settings.Additionally, Canary effectively guides off-the-shelf language models to generate significantly more prosocial responses.Our work highlights the promise and importance of creating and steering conversational AI to be socially responsible.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Data generation derived from crowdsourcing may favor majority opinions.
“Data generation derived from crowdsourcing, for instance, may favor majority opinions, as Kim et al. (2022) point out in their creation of an inherently subjective social norm dataset, based on the Social Chemistry dataset that Forbes et al. (2020) acknowledge to represent primarily English-speaking, North American norms.”
From Bias and Fairness in Large Language Models: A Survey (2024) · cites this paper as Kim et al. 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Social exclusion decreases prosocial behavior
Twenge, J. M., Baumeister, R. F., DeWall, C. N., Ciarocco, N. J., & Bartels, J. M. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.1.56 · Cited by 1,271 · see citing works
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784 · see citing works
The design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777 · see citing works
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
Internet-Augmented Dialogue Generation
Komeili, M., Shuster, K., & Weston, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.579 · Cited by 192 · see citing works
Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
Dinan, E., Humeau, S., Chintagunta, B., & Weston, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1461 · Cited by 161 · see citing works
Empathy-based counterspeech can reduce racist hate speech in a social media field experiment
Hangartner, D., Gennaro, G., Alasiri, S., Bahrich, N., Bornhoft, A., Boucher, J., Demirci, B. B., Derksen, L., Hall, A., Jochum, M., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2116310118 · Cited by 144 · see citing works
Bot-Adversarial Dialogue for Safe Conversational Agents
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.235 · Cited by 64 · see citing works
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems
Dinan, E., Abercrombie, G., Bergman, A., Spruit, S., Hovy, D., Boureau, Y.-L., & Rieser, V. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.284 · Cited by 32 · see citing works
Flagged first, then by citations
Research works printed but not linked: 51
- Maurianne Adams, Warren J Blumenfeld, Rosie Cas- tañeda, Heather W Hackman, Madeline L Peters, and Ximena Zúñiga. 2000. Readings for diversity and social justice. Psychology Press.
- Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021. Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts. In EMNLP.
- C. Daniel Batson and Adam A. Powell. 2003. Altruism and Prosocial Behavior. In Handbook of Psychol- ogy, 5th edition.
- Roy F. Baumeister and Brad J. Bushman. 2017. Social Psychology and Human Nature, 4th edition. Cen- gage Learning.
- Paul Bloom. 2010. How do Morals Change? Nature, 464(7288):490-490.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In NeurIPS.
- Frances S Chen, Julia A Minson, and Zakary L Tormala. 2010. Tell Me More: The Effects of Expressed Interest on Receptiveness during Dia- log. Journal of Experimental Social Psychology, 46(5):850-853.
- Herbert H Clark and Susan E Brennan. 1991. Ground- ing in communication. In Perspectives on socially shared cognition., pages 127-149. American Psy- chological Association.
- Kate Crawford. 2021. Atlas of AI. Yale University Press. · DOI
- Leslie A DeChurch and Michelle A Marks. 2001. Max- imizing the benefits of task conflict: The role of con- flict management. International Journal of Conflict Management.
- Dorottya Demszky, Dana Movshovitz-Attias, Jeong- woo Ko, Alan Cowen, Gaurav Nemade, and Su- jith Ravi. 2020. GoEmotions: A Dataset of Fine- Grained Emotions. In ACL.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In NAACL.
- Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of Wikipedia: Knowledge-Powered Conversational Agents. In ICLR.
- Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social Chem- istry 101: Learning to Reason about Social and Moral Norms. In EMNLP.
- Sam Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. RealToxici- tyPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of EMNLP.
- Jonathan Haidt, Silvia Helena Koller, and Maria G Dias. 1993. Affect, culture, and morality, or is it wrong to eat your dog? Journal of personality and social psychology, 65(4):613.
- John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research, 77(1):81- 112.
- Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning AI With Shared Human Values. In ICLR.
- Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. 2021. Understanding by Understanding Not: Modeling Negation in Language Models. In NAACL.
- Karen Huang, Michael Yeomans, Alison Wood Brooks, Julia Minson, and Francesca Gino. 2017. It doesn't Hurt to Ask: Question-asking Increases Lik- ing. Journal of personality and social psychology, 113(3):430.
- Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Le Bras Ronan, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: Towards Machine Ethics and Norms. arXiv preprint arXiv:2110.07574. · arXiv
- Mucahid Kutlu, Tyler McDonnell, Tamer Elsayed, and Matthew Lease. 2020. Annotator rationales for la- beling tasks in crowdsourcing. The journal of artifi- cial intelligence research, 69:143-189.
- Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards Emotional Support Dialog Systems. In ACL.
- Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-Shot Bot: Prompt- Based Learning for Dialogue Systems. arXiv preprint arXiv:2110.08118. · arXiv
- Shikib Mehri, Jinho Choi, Luis Fernando D'Haro, Jan Deriu, Maxine Eskenazi, Milica Gasic, Kallirroi Georgila, Dilek Hakkani-Tur, Zekang Li, Verena Rieser, Samira Shaikh, David Traum, Yi-Ting Yeh, Zhou Yu, Yizhe Zhang, and Chen Zhang. 2022. Re- port from the NSF future directions workshop on automatic evaluation of dialog: Research directions and challenges.
- A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bor- des, D. Parikh, and J. Weston. 2017. ParlAI: A Dia- log Research Software Platform. arXiv:1705.06476. · arXiv
- Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M Khapra. 2018. Towards Exploiting Back- ground Knowledge for Building Conversation Sys- tems. In EMNLP.
- Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. In NAACL.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Lan- guage Models are Unsupervised Multitask Learners. OpenAI blog, 1(8):9.
- M Afzalur Rahim. 2002. Toward a theory of manag- ing organizational conflict. International journal of conflict management.
- Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards Empathetic Open- domain Conversation Models: A New Benchmark and Dataset. In ACL.
- Carl R. Rogers. 1946. Significant Aspects of Client-centered Therapy. American Psychologist, 1(10):415.
- Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Im- plications of Language. In ACL.
- Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. In NAACL.
- Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills. In ACL.
- Miriah Steiger, Timir J Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. 2021. The psy- chological Well-Being of content moderators: The emotional labor of commercial moderation and av- enues for improving support. In CHI.
- Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2022. On the Safety of Con- versational Models: Taxonomy, Dataset, and Bench- mark. In Findings of ACL.
- Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2021. A Word on Machine Ethics: A Response to Jiang et al.(2021). arXiv preprint arXiv:2111.04158. · arXiv
- Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. LaMDA: Language Models for Dialog Applications. arXiv preprint arXiv:2201.08239. · arXiv
- Megan Ung, Jing Xu, and Y-Lan Boureau. 2021. Saferdialogues: Taking feedback gracefully af- ter conversational safety failures. arXiv preprint arXiv:2110.07518. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Rea- soning in Large Language Models. arXiv preprint arXiv:2201.11903. · arXiv
- Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and Social Risks of Harm from Language Models. arXiv preprint arXiv:2112.04359. · arXiv
- Anuradha Welivita and Pearl Pu. 2020. A Taxonomy of Empathetic Response Intents in Human Social Con- versations. In COLING.
- Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Ja- son Weston, and Emily Dinan. 2020. Recipes for Safety in Open-domain Chatbots. arXiv preprint arXiv:2010.07079. · arXiv
- Iris Marion Young. 2014. Five faces of oppression. Re- thinking power, pages 174-195.
- Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing Dialogue Agents: I Have a Dog, Do You Have Pets Too? In ACL.
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. OPT: Open Pre-trained Transformer Lan- guage Models. arXiv preprint arXiv:2205.01068. · arXiv
- Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DialoGPT : Large- Scale Generative Pre-training for Conversational Re- sponse Generation. In ACL: System Demonstra- tions.
- Pei Zhou, Pegah Jandaghi, Hyundong Cho, Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2021a. Probing Commonsense Explanation in Dialogue Response Generation. In Findings of EMNLP.
- Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah A Smith. 2021b. Challenges in Automated Debiasing for Toxic Language Detec- tion. In EACL.
- Caleb Ziems, Jane A Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems. In ACL.
Cited by
Sorted by citations
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 665 · see citing works
Mirages. On Anthropomorphism in Dialogue Systems
Abercrombie, G., Curry, A., Dinkar, T., Rieser, V., & Talat, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.290 · Cited by 83 · see citing works
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Kim, H., Hessel, J., Jiang, L., West, P., Lu, X., Yu, Y., Zhou, P., Bras, R., Alikhani, M., Kim, G., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.799 · Cited by 63 · see citing works
Self-citationSarkar, S., Gaur, M., Chen, L. K., Garg, M., & Srivastava, B. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1229805 · Cited by 38 · see citing works
KoSBI: A Dataset for Mitigating Social Bias Risks Towards Safer Large Language Model Applications
Lee, H., Hong, S., Park, J., Kim, T., Kim, G., & Ha, J.-W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track) · doi:10.18653/v1/2023.acl-industry.21 · Cited by 15 · see citing works
Self-citationJang, J., Boo, M., & Kim, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.838 · Cited by 13 · see citing works
Self-citationCoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
Yu, E., Li, J., Liao, M., Wang, S., Gao, Z., Mi, F., & Hong, L. (2024) · arXiv · doi:10.48550/arxiv.2406.17626 · Cited by 7 · see citing works
Paciello, M., Corbelli, G., Sportelli, C., Cicirelli, P. G., & D’Errico, F. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-08382-7 · Cited by 1 · see citing works
- Authors
- Kim, H., Yu, Y., Jiang, L., Lu, X., Khashabi, D., Kim, G., Choi, Y., & Sap, M.
- Affiliations
- Seoul National University (KR)
- Corresponding author
- Kim, Hyunwoo
- ORCID
- Venue
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Published
- 2022
- Pages
- 4005-4029
- OpenAlex
- W4385573018
- Cited by
- 80
- References
- 12
- Citations by year
- 2026: 15 · 2025: 20 · 2024: 14 · 2023: 28 · 2022: 3
- FWCI
- 8.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Allen Institute for Artificial Intelligence
- Ministry of Science and ICT, South Korea (No.2022-0-00156)
- Defense Advanced Research Projects Agency (N66001, N66001-19-2-4031, N66001-19)
- Institute for Information and Communications Technology Promotion (2022-)
- Naval Information Warfare Center Pacific
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- chatbots, toxic content, rules of thumb, human-AI collaboration, dialogue generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
