CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
Open accessAbstract
As large language models (LLMs) constantly evolve, ensuring their safety remains a critical research problem. Previous red-teaming approaches for LLM safety have primarily focused on single prompt attacks or goal hijacking. To the best of our knowledge, we are the first to study LLM safety in multi-turn dialogue coreference. We created a dataset of 1,400 questions across 14 categories, each featuring multi-turn coreference safety attacks. We then conducted detailed evaluations on five widely used open-source LLMs. The results indicated that under multi-turn coreference safety attacks, the highest attack success rate was 56% with the LLaMA2-Chat-7b model, while the lowest was 13.9% with the Mistral-7B-Instruct model. These findings highlight the safety vulnerabilities in LLMs during dialogue coreference interactions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CoSafe revealed that even advanced agents had vulnerabilities, such as falling for coreference-based attacks.
“CoSafe revealed that even advanced agents had vulnerabilities, such as falling for coreference-based attacks (where a user refers to something ambiguously to bypass filters).”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [110]
CoSafe evaluates conversation agents on adversarial prompts designed to trick them into breaking safety rules.
“Yu et al. introduce CoSafe [110] to evaluate conversation agents on adversarial prompts designed to trick them into breaking safety rules (e.g., a user subtly asks for self-harm advice or illicit instructions).”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [110]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 2× · see citing works
“Some studies (Shaikh et al., 2023; Wang et al., 2023; Lin et al., 2022) attempted to attack LLMs from the content perspective by defining and exploring various harmful categories.”
BackgroundEnd-to-end Neural Coreference Resolution
Lee, K., He, L., Lewis, M., & Zettlemoyer, L. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1018 · Cited by 869 · cited 1× · see citing works
“Various methods have been studied to solve this problem, including rule-based (Lee et al., 2011), statistical (Durrett and Klein, 2013; Björkelund and Kuhn, 2014), and neural approaches (Clark and Manning, 2016; Lee et al., 2017; Bohnet et al., 2023).”
MethodGender bias in coreference resolution: Evaluation and debiasing methods
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2003 · Cited by 684 · cited 2× · see citing works
“Furthermore, existing studies have created various red-teaming datasets to assess the safety of LLMs in different scenarios, such as bias (Zhao et al., 2018), discrimination (Shaikh et al., 2023), and privacy breaches (Mireshghallah et al., 2024a).”
BackgroundMethodGender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 2× · see citing works
“Type Purpose WinoGender (Rudinger et al., 2018a) 720 Prompt Evaluate gender bias in coreference resolution.”
BackgroundMethodDeep Reinforcement Learning for Mention-Ranking Coreference Models
Clark, K., & Manning, C. D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1245 · Cited by 383 · cited 1× · see citing works
“Various methods have been studied to solve this problem, including rule-based (Lee et al., 2011), statistical (Durrett and Klein, 2013; Björkelund and Kuhn, 2014), and neural approaches (Clark and Manning, 2016; Lee et al., 2017; Bohnet et al., 2023).”
MethodAre Large Pre-Trained Language Models Leaking Your Personal Information?Preprint
Huang, J., Shao, H., & Chang, K. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.12628 · Cited by 315 · cited 1× · see citing works
“Others focused on the types of attacks, such as dialogue safety attacks (Dinan et al., 2019; Ung et al., 2022; Kim et al., 2022), privacy attacks (Huang et al., 2022; Mireshghallah et al., 2024b), and human value alignment (Santurkar et al., 2023; Perez et al., 2023; Jin et al., 2024).”
BackgroundEasy Victories and Uphill Battles in Coreference Resolution
Durrett, G., & Klein, D. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1203 · Cited by 279 · cited 1× · see citing works
“Various methods have been studied to solve this problem, including rule-based (Lee et al., 2011), statistical (Durrett and Klein, 2013; Björkelund and Kuhn, 2014), and neural approaches (Clark and Manning, 2016; Lee et al., 2017; Bohnet et al., 2023).”
MethodDo-Not-Answer: A Dataset for Evaluating Safeguards in LLMsPreprint
Wang, Y., Li, H., Han, X., Nakov, P., & Baldwin, T. (2023) · arXiv.org · doi:10.48550/arxiv.2308.13387 · Cited by 236 · cited 3× · see citing works
“Some studies (Shaikh et al., 2023; Wang et al., 2023; Lin et al., 2022) attempted to attack LLMs from the content perspective by defining and exploring various harmful categories.”
BackgroundMethodDiscovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.847 · Cited by 194 · see citing works
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186 · cited 1× · see citing works
“These categories are defined in the Beavertail (Ji et al., 2023), which is derived from studies on the harmful content generation of LLMs (Ganguli et al., 2022; Parrish et al., 2022; Rauh et al., 2022).”
BackgroundProsocialDialog: A Prosocial Backbone for Conversational AgentsPreprint
Kim, H., Yu, Y., Jiang, L., Lu, X., Khashabi, D., Kim, G., Choi, Y., & Sap, M. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.12688 · Cited by 179 · cited 2× · see citing works
“Others focused on the types of attacks, such as dialogue safety attacks (Dinan et al., 2019; Ung et al., 2022; Kim et al., 2022), privacy attacks (Huang et al., 2022; Mireshghallah et al., 2024b), and human value alignment (Santurkar et al., 2023; Perez et al., 2023; Jin et al., 2024).”
BackgroundMethodBuild it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
Dinan, E., Humeau, S., Chintagunta, B., & Weston, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1461 · Cited by 161 · cited 1× · see citing works
“Others focused on the types of attacks, such as dialogue safety attacks (Dinan et al., 2019; Ung et al., 2022; Kim et al., 2022), privacy attacks (Huang et al., 2022; Mireshghallah et al., 2024b), and human value alignment (Santurkar et al., 2023; Perez et al., 2023; Jin et al., 2024).”
BackgroundWhen to Make Exceptions: Exploring Language Models as Accounts of Human Moral JudgmentPreprint
Jin, Z., Levine, S., Gonzalez, F., Kamal, O., Sap, M., Sachan, M., Mihalcea, R., Tenenbaum, J. B., & Schölkopf, B. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.01478 · Cited by 147 · cited 1× · see citing works
“Others focused on the types of attacks, such as dialogue safety attacks (Dinan et al., 2019; Ung et al., 2022; Kim et al., 2022), privacy attacks (Huang et al., 2022; Mireshghallah et al., 2024b), and human value alignment (Santurkar et al., 2023; Perez et al., 2023; Jin et al., 2024).”
BackgroundAre Large Pre-Trained Language Models Leaking Your Personal Information?
Huang, J., Shao, H., & Chang, K. C. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.148 · Cited by 135 · see citing works
Björkelund, A., & Kuhn, J. (2014) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p14-1005 · Cited by 132 · cited 1× · see citing works
“Various methods have been studied to solve this problem, including rule-based (Lee et al., 2011), statistical (Durrett and Klein, 2013; Björkelund and Kuhn, 2014), and neural approaches (Clark and Manning, 2016; Lee et al., 2017; Bohnet et al., 2023).”
MethodWhose Opinions Do Language Models Reflect?
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2303.17548 · Cited by 114 · cited 1× · see citing works
“Others focused on the types of attacks, such as dialogue safety attacks (Dinan et al., 2019; Ung et al., 2022; Kim et al., 2022), privacy attacks (Huang et al., 2022; Mireshghallah et al., 2024b), and human value alignment (Santurkar et al., 2023; Perez et al., 2023; Jin et al., 2024).”
BackgroundBeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset
Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Zhang, C., Sun, R., Wang, Y., & Yang, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.04657 · Cited by 88 · cited 5× · see citing works
“Red Teaming Language Models Ganguli et al. (2022) created a dataset called AnthropicRedTeam to identify, quantify, and mitigate potentially harmful outputs from LLMs. Ji et al. (2023) sampled from AnthropicRedTeam to build a dataset called BeaverTails for QA contexts.”
BackgroundMethodInfluentialProsocialDialog: A prosocial backbone for conversational agents
Kim, H., Yu, Y., Jiang, L., Lu, X., Khashabi, D., Kim, G., Choi, Y., & Sap, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.267 · Cited by 80 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76 · cited 3× · see citing works
“HH-RLHF (Ganguli et al., 2022) 38 , 961 Dialogue Analyse how people red-team LLMs. HarmfulQ (Shaikh et al., 2023) 200 Prompt Machine-written prompts to evaluate LLM safety.”
BackgroundCharacteristics of Harmful Text: Towards Rigorous Benchmarking of Language ModelsPreprint
Rauh, M., Mellor, J. F. J., Uesato, J., Huang, P.-S., Welbl, J., Weidinger, L., Dathathri, S., Glaese, A., Irving, G., Gabriel, I., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.08325 · Cited by 72 · cited 1× · see citing works
“These categories are defined in the Beavertail (Ji et al., 2023), which is derived from studies on the harmful content generation of LLMs (Ganguli et al., 2022; Parrish et al., 2022; Rauh et al., 2022).”
BackgroundDiscovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53 · cited 1× · see citing works
“Others focused on the types of attacks, such as dialogue safety attacks (Dinan et al., 2019; Ung et al., 2022; Kim et al., 2022), privacy attacks (Huang et al., 2022; Mireshghallah et al., 2024b), and human value alignment (Santurkar et al., 2023; Perez et al., 2023; Jin et al., 2024).”
BackgroundCoreference Resolution through a seq2seq Transition-Based System
Bohnet, B., Alberti, C., & Collins, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00543 · Cited by 35 · cited 1× · see citing works
“Various methods have been studied to solve this problem, including rule-based (Lee et al., 2011), statistical (Durrett and Klein, 2013; Björkelund and Kuhn, 2014), and neural approaches (Clark and Manning, 2016; Lee et al., 2017; Bohnet et al., 2023).”
MethodSaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures
Ung, M., Xu, J., & Boureau, Y.-L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.447 · Cited by 24 · cited 2× · see citing works
“SaFeRDialogues (Ung et al., 2022) 7 , 881 Dialogue Recover from safety failures in LLM conversations.”
BackgroundMireshghallah, N., Kim, H., Zhou, X., Tsvetkov, Y., Sap, M., Shokri, R., & Choi, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.17884 · Cited by 14 · cited 1× · see citing works
“Furthermore, existing studies have created various red-teaming datasets to assess the safety of LLMs in different scenarios, such as bias (Zhao et al., 2018), discrimination (Shaikh et al., 2023), and privacy breaches (Mireshghallah et al., 2024a).”
Background
Flagged first, then by citations
3 of the 28 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Ju- rafsky. 2011. Stanford's multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifteenth Conference on Compu- tational Natural Language Learning: Shared Task, pages 28-34, Portland, Oregon, USA. Association for Computational Linguistics.
- Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018b. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8-14, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
Cited by
Sorted by citations
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · see citing works
“For example, the HELM benchmark [51] for enterprises was proposed to include domain-specific prompts and metrics (such as accuracy on financial jargon or compliance in responses) for fields like finance and law.”
Zhang, J., Xia, Y., Zhong, H., Lu, W., Deng, Q., & Wan, C. (2025) · Mathematics · doi:10.3390/math14010090 · Cited by 4 · see citing works
Human preferences are susceptible to covertly misaligned AI advice.
Sabour, S., Liu, J. M., Liu, S., Yao, C. Z., Cui, S., Zhang, W., Zhang, X., Cao, Y.-R., Bhat, A., Guan, J., et al. (2026) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2600684123 · Cited by 3 · see citing works
Kasneci, G., & Kasneci, E. (2026) · AI and Ethics · doi:10.1007/s43681-026-01132-0 · Cited by 1 · see citing works
Breaking Bad Intentions: Identifying Semantic Drift in Multi-Turn Language Model InputsPreprint
Amo, E., Cummings, M. L., & Chen, W. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7023228
REMEDy: a dataset for rationale extraction and span-based moderation of dialogue prompts
Piano, L., Battistin, C., den Abeele, J. V., & Pompianu, L. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12381-9
O., E. F. T. (2026) · Research Square · doi:10.21203/rs.3.rs-11210583/v1
The Illusion of Success: Procedure-Aware Evaluation is Critical for LLM Agents
Cao, H., Driouich, I., & Thomas, E. M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37682-4_29 · Cited by 5 · see citing works
- Authors
- Yu, E., Li, J., Liao, M., Wang, S., Gao, Z., Mi, F., & Hong, L.
- Affiliations
- Hong Kong Polytechnic University (HK)
- Huawei Technologies (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 17494-17508
- arXiv
- 2406.17626
- OpenAlex
- W4404782832
- Cited by
- 7
- References
- 28
- Citations by year
- 2026: 6 · 2025: 1
- FWCI
- 2.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- dx.doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Natural Science Foundation of China
- Hong Kong Polytechnic University
- Innovation and Technology Fund (PRP/047/22FX)
- Young Scientists Fund
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language model safety, multi-turn dialogue, coreference resolution, red teaming, attack dataset, Mistral-7B-Instruct, open-source LLMs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
