Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
Open accessAbstract
The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.) introduces a new set of inference-time privacy risks: LLMs are fed different types of information from multiple sources in their inputs and are expected to reason about what to share in their outputs, for what purpose and with whom, within a given context. In this work, we draw attention to the highly critical yet overlooked notion of contextual privacy by proposing ConfAIde, a benchmark designed to identify critical weaknesses in the privacy reasoning capabilities of instruction-tuned LLMs. Our experiments show that even the most capable models such as GPT-4 and ChatGPT reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. This leakage persists even when we employ privacy-inducing prompts or chain-of-thought reasoning. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs can be sensitive to irrelevant context, exhibiting context leakage.
“Prior work has shown that LLMs can be sensitive to irrelevant context, exhibiting context leakage (Mireshghallah et al., 2023; Gupta et al., 2024; Hui et al., 2024), and display sycophantic behavior with responses favoring perceived user preferences rather than objective evidence (Sharma et al., 2023; Perez et al., 2023; Fanous et al., 2025).”
From PersistBench: When Should Long-Term Memories Be Forgotten by LLMs? (2026) · cites this paper as Mireshghallah et al., 2023
Section 4.3 reports that LLMs leak secrets from their input context into responses, including meeting summaries and replies where the recipient should not learn the secret. This supports context leakage in the sense of inappropriate disclosure, but the paper does not establish the broader claim that LLMs are sensitive to irrelevant context in general. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 42
- Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, oct 2016. doi: 10.1145/2976749.2978318. URL https://doi.org/10.1145%2F2976749.2978318. · DOI · Paper page
- Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. · arXiv
- Torben Braüner, Patrick Blackburn, and Irina Polyanskaya. Being deceived: Information asymmetry in second-order false belief tasks. Topics in cognitive science, 2019. URL https://api.semanticscholar.org/CorpusID:150019723.
- Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 2280-2292, 2022.
- Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. · arXiv
- Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter. Can language models be instructed to protect personal information? arXiv preprint arXiv:2310.02224, 2023. · arXiv
- Malinda J Colwell, Kimberly Corson, Anuradha Sastry, and Holly Wright. Secret keepers: children's theory of mind and their conception of secrecy. Early Child Development and Care, 186(3):369-381, 2016.
- Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochastic parrots: Differentially private prompt learning for large language models. arXiv preprint arXiv:2305.15594, 2023. · arXiv
- Chris D. Frith and Uta Frith. Interacting minds-a biological basis. Science, 286 (5445):1692-1695, 1999. URL https://api.semanticscholar.org/CorpusID:16546828.
- John Hart, J. Gratch, and Stacy Marsella. How virtual reality training can win friends and influence people. 2013. URL https://api.semanticscholar.org/CorpusID:146359723.
- Paul M Heider, Jihad S Obeid, and Stéphane M Meystre. A comparative analysis of speed and accuracy for three off-the-shelf de-identification tools. AMIA Summits on Translational Science Proceedings, 2020:241, 2020.
- Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. · arXiv
- Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ro- nan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. Soda: Million-scale dialogue distillation with social commonsense contextualization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12930-12949, 2023a. URL https://aclanthology.org/2023.emnlp-main.799.
- Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. Fantom: A benchmark for stress-testing machine theory of mind in interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14397-14413, 2023b. URL https://aclanthology.org/2023.emnlp-main.890.
- Nadin Kökciyan. Privacy management in agent-based social networks. In AAAI, pp. 4299-4300, 2016.
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020. URL https://api.semanticscholar.org/CorpusID:218869575.
- Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679, 2021. · arXiv
- David J Martin, Daniel Kifer, Ashwin Machanavajjhala, Johannes Gehrke, and Joseph Y Halpern. Worst-case background knowledge for privacy-preserving data publishing. In 2007 IEEE 23rd International Conference on Data Engineering, pp. 126-135. IEEE, 2006.
- Rachel I. McDonald, Jessica M. Salerno, Katharine H. Greenaway, and Michael L. Slepian. Motivated secrecy: Politics, relationships, and regrets. Current Opinion in Organ Transplantation, 6:61-78, 2020. URL https://api.semanticscholar.org/CorpusID:181865952.
- Helen Nissenbaum. Privacy in Context: Technology, Policy, and the Integrity of Social Life. Stanford University Press, 2009.
- OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023a. URL https://api. semanticscholar.org/CorpusID:257532815. · arXiv
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
- Ashwinee Panda, Tong Wu, Jiachen T Wang, and Prateek Mittal. Differentially private in-context learning. arXiv preprint arXiv:2305.01639, 2023. · arXiv
- Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. · arXiv
- David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515-526, 1978.
- Aman Priyanshu, Supriti Vijay, Ayush Kumar, Rakshit Naidu, and Fatemehsadat Mireshghallah. Are chatbots ready for privacy-sensitive applications? an investigation into input regurgitation and prompt-induced sanitization. arXiv preprint arXiv:2305.15008, 2023. · arXiv
- Zhenting Qi, Hanlin Zhang, Eric Xing, Sham Kakade, and Himabindu Lakkaraju. Follow my instruction and spill the beans: Scalable data extraction from retrieval-augmented generation systems. arXiv preprint arXiv:2402.17840, 2024. · arXiv
- Jessica M Salerno and Michael L Slepian. Morality, punishment, and revealing other people's secrets. Journal of Personality and Social Psychology, 122(4):606, 2022.
- Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? on the limits of social intelligence in large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3762-3780, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.248.
- Shalom H Schwartz. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2(1):11, 2012.
- Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Minding language models'(lack of) theory of mind: A plug-and-play multi-character belief tracker. arXiv preprint arXiv:2306.00924, 2023. · arXiv
- Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763, 2023. · arXiv
- Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. Selective differential privacy for language modeling. arXiv preprint arXiv:2108.12944, 2021. · arXiv
- Yan Shvartzshnaider, Zvonimir Pavlinovic, Ananth Balashankar, Thomas Wies, Lakshminarayanan Subramanian, Helen Nissenbaum, and Prateek Mittal. Vaccine: Using contextual integrity for data leakage detection. In The World Wide Web Conference, pp. 1702-1712, 2019.
- Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-preserving in-context learning with differentially private few-shot generation. arXiv preprint arXiv:2309.11765, 2023. · arXiv
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
- Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
- Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023. · arXiv
- Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500, 2021. · arXiv
- Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. (inthe)wildchat: 570k chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Bl8u7ZRlbM.
- Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas Davidson, Jena D. Hwang, Swabha Swayamdipta, and Maarten Sap. COBRA frames: Contextual reasoning about effects and harms of offensive statements. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 6294-6315, 2023.
Cited by
Sorted by citations
AirGapAgent: Protecting Privacy-Conscious Conversational Agents
Bagdasarian, E., Yi, R., Ghalebikesabi, S., Kairouz, P., Gruteser, M., Oh, S., Balle, B., & Ramage, D. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3690350 · Cited by 28 · see citing works
“A similar problem was recently studied in [36], where a multi-tier benchmark for assessing the context-dependent privacy awareness of LLMs was proposed.”
Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots
Zhou, J., Xu, E., Wu, Y., & Li, T. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713701 · Cited by 17 · see citing works
CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
Yu, E., Li, J., Liao, M., Wang, S., Gao, Z., Mi, F., & Hong, L. (2024) · arXiv · doi:10.48550/arxiv.2406.17626 · Cited by 7 · see citing works
“Furthermore, existing studies have created various red-teaming datasets to assess the safety of LLMs in different scenarios, such as bias (Zhao et al., 2018), discrimination (Shaikh et al., 2023), and privacy breaches (Mireshghallah et al., 2024a).”
BackgroundEIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
Liao, Z., Mo, L., Xu, C., Kang, M., Zhang, J., Xiao, C., Tian, Y., Li, B., & Sun, H. (2024) · arXiv · doi:10.48550/arxiv.2409.11295 · Cited by 1 · see citing works
PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?
Pulipaka, S., Chen, O., Sharma, M., Bajwa, T. S., Raina, V., & Sheth, I. (2026) · arXiv · doi:10.48550/arxiv.2602.01146
Roesner, F., & Kohno, T. (2026) · arXiv · doi:10.48550/arxiv.2609.17817
- Authors
- Mireshghallah, N., Kim, H., Zhou, X., Tsvetkov, Y., Sap, M., Shokri, R., & Choi, Y.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Carnegie Mellon University (US)
- National University of Singapore (SG)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.17884
- OpenAlex
- W4388092128
- Cited by
- 14
- References
- 0
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2125201, 2142739, 2203097, CAREER)
- Alfred P. Sloan Foundation
- Office of the Director of National Intelligence
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (HR001120C0124, HR00112020054, N66001-19-2-4031, N66001-19)
- Intelligence Advanced Research Projects Activity
- Naval Information Warfare Center Pacific
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- large language models, privacy, contextual integrity, contextual privacy, AI assistants, chain-of-thought reasoning, theory of mind, prompt engineering, GPT-4, ChatGPT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
