Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs
Open accessAbstract
The rapid proliferation of frontier model agents promises significant societal advances but also raises concerns about systemic risks arising from unsafe interactions. Collusion to the disadvantage of others has been identified as a central form of undesirable agent cooperation. The use of information hiding (steganography) in agent communications could render such collusion practically undetectable. This underscores the need for investigations into the possibility of such behaviours emerging and the robustness corresponding countermeasures. To investigate this problem we design two approaches -- a gradient-based reinforcement learning (GBRL) method and an in-context reinforcement learning (ICRL) method -- for reliably eliciting sophisticated LLM-generated linguistic text steganography. We demonstrate, for the first time, that unintended steganographic collusion in LLMs can arise due to mispecified reward incentives during training. Additionally, we find that standard mitigations -- both passive oversight of model outputs and active mitigation through communication paraphrasing -- are not fully effective at preventing this steganographic communication. Our findings imply that (i) emergence of steganographic collusion is a plausible concern that should be monitored and researched, and (ii) preventing emergence may require innovation in mitigation techniques.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These steganographic capabilities emerged without explicit training in steganography, instead leveraging general language processing and reasoning abilities.
“Crucially, these capabilities emerged without explicit training in steganography, instead leveraging general language processing and reasoning abilities - a phenomenon also observed by Mathew et al. (2024).”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Mathew et al., 2024
Sections 4.1 and 5.1 describe steganographic communication emerging in a candidate-screening task when models were optimized for prediction accuracy while avoiding an overseer’s penalty. That supports the observation of unintended emergence, but the paper does not establish that it arose specifically from general language-processing and reasoning abilities. Section 5.2 also describes separate experiments in which models were explicitly prompted to develop steganography strategies, so the claim should not be applied to all its results. (LLM, conf=0.83)
Experimental — accuracy not yet measuredOnly a small number of works have considered collusion in the context of more advanced AI agents, such as those powered by LLMs, and this remains a highly under-explored area.
“Only a small number of works, however, have considered collusion in the context of more advanced AI agents, such as those powered by LLMs (Fish et al., 2024; Mathew et al., 2024; Motwani et al., 2024; OpenAI, 2023c), and this remains a highly under-explored area.”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Mathew et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Research works printed but not linked: 28
- Martín Abadi and David G Andersen. 2016. Learning to protect communications with adversarial neural cryptography. arXiv preprint arXiv:1610.06918. · arXiv
- Lori Andrews and Hannah Bucher. 2022. Automating Discrimination: AI Hiring Practices and Gender In- equality. Cardozo Law Review, 44.
- Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926. · arXiv
- Xinying Chen, Jianping An, Zehui Xiong, Chengwen Xing, Nan Zhao, F. Richard Yu, and Arumugam Nal- lanathan. 2023. Covert communications: A compre- hensive survey. IEEE Communications Surveys & Tutorials, 25(2):1173-1198. · DOI
- Zhisheng Chen. 2023. Ethics and discrimination in artificial intelligence-enabled recruitment practices. Humanities and Social Sciences Communications, 10(1):1-12. Publisher: Palgrave. · DOI · Paper page
- Nicholas Goldowsky-Dill, Bilal Chughtai, Stefan Heimersheim, and Marius Hobbhahn. 2025. Detect- ing strategic deception using linear probes. arXiv preprint arXiv:2502.03407. · arXiv
- Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. 2024. AI Control: Improving Safety Despite Intentional Subversion. arXiv preprint. ArXiv:2312.06942 [cs. · DOI · Paper page
- Danny Halawi, Alexander Wei, Eric Wallace, Tony T. Wang, Nika Haghtalab, and Jacob Steinhardt. 2024. Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation. arXiv preprint. ArXiv:2406.20053 [cs]. · DOI · Paper page
- Serhii Havrylov and Ivan Titov. 2017. Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols. In Ad- vances in Neural Information Processing Systems (NeurIPS), volume 30.
- Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
- Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, and 1 others. 2023. Measuring faithful- ness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702. · arXiv
- Angeliki Lazaridou and Marco Baroni. 2020. Emergent Multi-Agent Communication in the Deep Learning Era. arXiv preprint. ArXiv:2006.02419 [cs. · DOI
- Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program Induction by Rationale Genera- tion: Learning to Solve and Explain Algebraic Word Problems. Preprint, arXiv:1705.04146. · arXiv
- Robert Lockwood and Kevin Curran. 2017. Text based steganography. International Journal of Informa- tion Privacy, Security and Integrity, 3(2):134-153. Publisher: Inderscience Publishers. · DOI
- Mohammed Abdul Majeed, Rossilawati Sulaiman, Za- rina Shukur, and Mohammad Kamrul Hasan. 2021. A review on text steganography techniques. Mathe- matics, 9(21):2829. Number: 21 Publisher: Multidis- ciplinary Digital Publishing Institute.
- Sumeet Ramesh Motwani, Mikhail Baranchuk, Martin Strohmeier, Vijay Bolina, Philip H. S. Torr, Lewis Hammond, and Christian Schroeder de Witt. 2024. Secret Collusion Among Generative AI Agents. arXiv preprint. ArXiv:2402.07510 [cs]. · DOI
- F.A.P. Petitcolas, R.J. Anderson, and M.G. Kuhn. 1999. Information hiding-a survey. Proceedings of the IEEE, 87(7):1062-1078. · DOI
- Fabien Roger and Ryan Greenblatt. 2023. Prevent- ing Language Models From Hiding Their Reasoning. arXiv preprint. ArXiv:2310.18512 [cs]. · DOI
- V Sanh. 2019. DistilBERT, A Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint arXiv:1910.01108. · arXiv
- Christian Schroeder de Witt, Samuel Sokota, J. Zico Kolter, Jakob Foerster, and Martin Strohmeier. 2022. Perfectly Secure Steganography Using Minimum En- tropy Coupling. · DOI
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv preprint. ArXiv:1707.06347 [cs. · DOI · Paper page
- Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, and 1 others. 2025. Con- stitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18837. · arXiv
- Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Re- flexion: language agents with verbal reinforcement learning. In Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
- Samuel Sokota, Christian A. Schroeder De Witt, Max- imilian Igl, Luisa M. Zintgraf, Philip Torr, Mar- tin Strohmeier, Zico Kolter, Shimon Whiteson, and Jakob Foerster. 2022. Communicating via Markov Decision Processes. In Proceedings of the 39th In- ternational Conference on Machine Learning, pages 20314-20328. PMLR.
- Gemma Team. 2024a. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. · arXiv
- Llama Team. 2024b. The llama 3 herd of models. Preprint, arXiv:2407.21783. · arXiv
- Kyle Wagner, James A. Reggia, Juan Uriagereka, and Gerald S. Wilkinson. 2003. Progress in the Simu- lation of Emergent Communication and Language. Adaptive Behavior, 11(1):37-69. · DOI
- Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2024. Voyager: An open-ended embodied agent with large language models. Trans. Mach. Learn. Res., 2024.
- Authors
- Mathew, Y., Matthews, O., McCarthy, R., Velja, J., de Witt, C. S., Cope, D., & Schoots, N.
- Affiliations
- University of Oxford (GB)
- University College London (GB)
- ORCID
- Cope, Dylan
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2410.03768
- OpenAlex
- W4403928901
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Steganography and Watermarking Techniques
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- steganography, collusion, large language models, agent cooperation, information hiding, reward specification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
