The DOI matches its Crossref record.
Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers
Open accessAbstract
We study privacy leakage in the reasoning traces of large reasoning models used as personal agents.Unlike final outputs, reasoning traces are often assumed to be internal and safe.We challenge this assumption by showing that reasoning traces frequently contain sensitive user data, which can be extracted via prompt injections or accidentally leak into outputs.Through probing and agentic evaluations, we demonstrate that test-time compute approaches, particularly increased reasoning steps, amplify such leakage.While increasing the budget of those test-time compute approaches makes models more cautious in their final answers, it also leads them to reason more verbosely and leak more in their own thinking.This reveals a core tension: reasoning improves utility but enlarges the privacy attack surface.We argue that safety efforts must extend to the model's internal thinking, not just its outputs. 1 * Work done during an internship at Parameter Lab. 1 Code available at github.com/parameterlab/leaky_thoughts.AirGapAgent-R benchmark available at huggingface.co/datasets/parameterlab/leaky_thoughts. Prior work on contextual privacy
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Exposing reasoning traces in plaintext leaves proprietary systems vulnerable to model distillation by competitors and risks unmasking internal safety and refusal mechanisms or revealing harmful information.
“Exposing these reasoning traces in plaintext leaves proprietary systems highly vulnerable to model distillation by competitors (Muennighoff et al., 2025), and it risks unmasking internal safety and refusal mechanisms or revealing harmful information (Green et al., 2025;Mao et al., 2026).”
From Stealing Reasoning Traces from Proprietary LLM APIs (2026) · cites this paper as Green et al., 2025
The Ethical Considerations section says reasoning traces can contain unsafe content and are relatively easy to extract; the Safety of reasoning models section reports harmful content in traces even when final answers are safe. These passages support the risk of revealing harmful information, but they do not establish that exposed traces unmask internal safety or refusal mechanisms. The distillation claim is cited to a different source and is not assessed here. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 34
- Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vid- hisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Java- heripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Be- smira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, and 4 others. 2025. Phi-4-reasoning technical report. Preprint, arXiv:2504.21318. · arXiv
- Eugene Bagdasarian, Ren Yi, Sahra Ghalebikesabi, Pe- ter Kairouz, Marco Gruteser, Sewoong Oh, Borja Balle, and Daniel Ramage. 2024. Airgapagent: Pro- tecting privacy-conscious conversational agents. In Proceedings of the 2024 on ACM SIGSAC Confer- ence on Computer and Communications Security, CCS '24, page 3868-3882, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
- Yoav Benjamini and Yosef Hochberg. 1995. Control- ling the false discovery rate: A practical and pow- erful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289-300.
- Collin J. Brown. 2024. Improved neural word segmenta- tion for standard Tibetan. In Proceedings of the 2nd Workshop on Resources and Technologies for Indige- nous, Endangered and Lesser-resourced Languages in Eurasia (EURALI) @ LREC-COLING 2024, pages 12-17, Torino, Italia. ELRA and ICCL.
- Zhao Cheng, Diane Wan, Matthew Abueg, Sahra Ghalebikesabi, Ren Yi, Eugene Bagdasarian, Borja Balle, Stefan Mellem, and Shawn O'Banion. 2024. Ci-bench: Benchmarking contextual integrity of ai assistants on synthetic data. ArXiv preprint, abs/2409.13903. · arXiv
- Jonathan Evertz, Merlin Chlosta, Lea Schönherr, and Thorsten Eisenhofer. 2024. Whispers in the machine: Confidentiality in llm-integrated systems. ArXiv preprint, abs/2402.06922. · arXiv
- Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. ArXiv preprint, abs/2407.21783. · arXiv
- Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Jia Xu, Linjian Mo, and Min Zhang. 2025. Test-time computing: from system-1 thinking to system-2 thinking. ArXiv preprint, abs/2501.02497. · arXiv
- Wang Jiaqi, Li Xinliang, Liu Zhengliang, Wu Zihao, Zhong Tianyang, Shu Peng, Li Yiwei, Jiang Hanqi, Zhou Yifan, Chen Junhao, Ruan Wei, Pan Yi, Zhao Huaqin, Ma Chong, Yang Zhenyuan, Xu Shaochen, Zhang Ruidong, Dai Haixing, Zhao Lin, and 12 oth- ers. 2025. LLM Reasoning: from OpenAI O1 to DeepSeek R1. Working paper or preprint.
- Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2023. Propile: Probing privacy leakage in large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
- Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neu- big, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating multi- modal agents on realistic visual web tasks. In Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 881-905, Bangkok, Thailand. Associ- ation for Computational Linguistics. · DOI · Paper page
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi- cient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
- Haoran Li, Wenbin Hu, Huihao Jing, Yulin Chen, Qi Hu, Sirui Han, Tianshu Chu, Peizhao Hu, and Yangqiu Song. 2025b. Privaci-bench: Evaluating privacy with contextual integrity and legal compliance. ArXiv preprint, abs/2502.17041. · arXiv
- Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. 2024. Can llms keep a secret? testing pri- vacy implications of language models via contextual integrity theory. In The Twelfth International Con- ference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Helen Nissenbaum. 2004. Privacy as contextual in- tegrity. Washington Law Review, 79(1):119-157.
- David Pape, Sina Mavali, Thorsten Eisenhofer, and Lea Schönherr. 2024. Prompt obfuscation for large lan- guage models. ArXiv preprint, abs/2409.11026. · arXiv
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. · DOI · Paper page
- Haritz Puerto, Martin Gubri, Sangdoo Yun, and Seong Joon Oh. 2025. Scaling up membership in- ference: When and how attacks succeed on large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4165-4182, Albuquerque, New Mexico. Association for Computational Linguistics.
- Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, and Diyi Yang. 2024a. Privacylens: Evaluating privacy norm awareness of language models in action. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Pro- cessing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 -15, 2024.
- Yan Shvartzshnaider and Vasisht Duddu. 2025. Position: Contextual integrity washing for language models. ArXiv preprint, abs/2501.19173. · arXiv
- Li Siyan, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Yu. 2024. Pa- pillon: Privacy preservation from internet-based and local language model ensembles. ArXiv preprint, abs/2410.17127. · arXiv
- Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Ku- mar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. ArXiv preprint, abs/2408.03314. · arXiv
- Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2022. Will we run out of data? limits of llm scal- ing based on human-generated data. ArXiv preprint, abs/2211.04325. · arXiv
- Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zi- nan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2023. Decodingtrust: A comprehensive as- sessment of trustworthiness in GPT models. In Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Information Process- ing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023.
- Yeo Wei Jie, Ranjan Satapathy, Rick Goh, and Erik Cambria. 2024. How interpretable are reasoning ex- planations from prompting large language models? In Findings of the Association for Computational Lin- guistics: NAACL 2024, pages 2148-2164, Mexico City, Mexico. Association for Computational Lin- guistics. · DOI
- Jason Weston and Sainbayar Sukhbaatar. 2023. System 2 attention (is something you might need too). ArXiv preprint, abs/2311.11829. · arXiv
- Tong Wu, Chong Xiang, Jiachen T. Wang, Weichen Yu, Chawin Sitawarin, Vikash Sehwag, and Prateek Mit- tal. 2025b. Does more inference-time compute really help robustness? arXiv preprint arXiv:2507.15974. · arXiv
- Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, and 1 others. 2025a. Towards large reasoning models: A survey of reinforced reasoning with large language models. ArXiv preprint, abs/2501.09686. · arXiv
- Haoran Xu, Baolin Peng, Hany Awadalla, Dongdong Chen, Yen-Chun Chen, Mei Gao, Young Jin Kim, Yunsheng Li, Liliang Ren, Yelong Shen, Shuohang Wang, Weijian Xu, Jianfeng Gao, and Weizhu Chen. 2025b. Phi-4-mini-reasoning: Exploring the limits of small reasoning language models in math. Preprint, arXiv:2504.21233. · arXiv
- Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. 2025. On protecting the data privacy of large language models (llms) and llm agents: A literature review. High-Confidence Computing, page 100300.
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
- Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Yuran Wang, Yong Ding, Yibo Zhang, Qi Zhang, and Xuanjing Huang. 2023. TextObfuscator: Making pre-trained language model a privacy protector via obfuscating word representations. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 5459-5473, Toronto, Canada. Association for Com- putational Linguistics. · DOI
- Xuechen Zhou, Shibani Santurkar, Deep Ganguli, Amanda Askell, David Krueger, Adam Lerer, Alex Kim, Aditya Malik, Miljan Martic, Cameron McK- innon, and Others. 2024. Deliberative alignment: Reasoning enables safer language models. ArXiv preprint, abs/2402.09353. · arXiv
- Xueyang Zhou, Guiyao Tie, Guowen Zhang, Weidong Wang, Zhigang Zuo, Di Wu, Duanfeng Chu, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2025b. Large reasoning models in agent scenarios: Explor- ing the necessity of reasoning capabilities. ArXiv preprint, abs/2503.11074. · arXiv
Cited by
Sorted by citations
Stealing Reasoning Traces from Proprietary LLM APIs
Panfilov, A., Schmotz, D., Shumailov, I., Beurer-Kellner, L., Schaeffer, J., Prabhu, A., Geiping, J., & Andriushchenko, M. (2026) · arXiv · doi:10.48550/arxiv.2608.09867
Metadata
- Authors
- Green, T., Gubri, M., Puerto, H., Yun, S., & Oh, S. J.
- Affiliations
- University of Mannheim (DE)
- Technische Universität Darmstadt (DE)
- NAVER Cloud (South Korea) (KR)
- Venue
- Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
- Published
- 2025
- Pages
- 26518-26540
- OpenAlex
- W4416035563
- Cited by
- 10
- References
- 0
- Citations by year
- 2026: 5 · 2025: 5
- FWCI
- 7.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- TUbilio (Technical University of Darmstadt) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Keywords
- privacy leakage, reasoning traces, large language models, personal agents, prompt injection, test-time compute, reasoning steps, contextual privacy, AI safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
