The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Operationalizing Contextual Integrity in Privacy-Conscious Assistants
Open accessAbstract
Advanced AI assistants combine frontier LLMs and tool access to autonomously perform complex tasks on behalf of users. While the helpfulness of such assistants can increase dramatically with access to user information including emails and documents, this raises privacy concerns about assistants sharing inappropriate information with third parties without user supervision. To steer information-sharing assistants to behave in accordance with privacy expectations, we propose to operationalize contextual integrity (CI), a framework that equates privacy with the appropriate flow of information in a given context. In particular, we design and evaluate a number of strategies to steer assistants' information-sharing actions to be CI compliant. Our evaluation is based on a novel form filling benchmark composed of human annotations of common webform applications, and it reveals that prompting frontier LLMs to perform CI-based reasoning yields strong results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current defenses often rely on training or prompting models to adhere to security policies, frequently implemented as vulnerable system prompts.
“Current defenses often rely on training or prompting models to adhere to security policies (Wallace et al., 2024; Ghalebikesabi et al., 2024), frequently implemented as vulnerable system prompts (Carlini et al., 2023).”
From Defeating Prompt Injections by Design (2026) · cites this paper as Ghalebikesabi et al., 2024
The Assistant design section (page unavailable) says privacy-conscious assistants can use prompting or fine-tuning, but its evaluation uses only prompting; Section 3.1 describes prompting an assistant to withhold inappropriate information. This supports prompting as a privacy defense, but not the broader claim that current defenses often rely on training. The vulnerable-system-prompts clause is cited to Carlini et al., not this paper. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340
Flagged first, then by citations
Research works printed but not linked: 33
- N. Abdi, X. Zhan, K. M. Ramokapane, and J. M. Such. "Privacy Norms for Smart Home Personal Assistants". In: CHI. 2021.
- M. Abdulhai, G. Serapio-Garcia, C. Crepy, D. Valter, J. Canny, and N. Jaques. "Moral foundations of large language models". In: arXiv preprint arXiv:2310.15337 (2023). · arXiv
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. "GPT-4 technical report". In: arXiv:2303.08774 (2023). · arXiv
- A. Arora, L.-A. Kaffee, and I. Augenstein. "Probing Pre-Trained Language Models for Cross- Cultural Differences in Values". In: Cross-Cultural Considerations in NLP@ EACL (2023), p. 114.
- E. Bagdasaryan, R. Yi, S. Ghalebikesabi, P. Kairouz, M. Gruteser, S. Oh, B. Balle, and D. Ramage. "Air Gap: Protecting Privacy-Conscious Conversational Agents". In: arXiv:2405.05175 (2024). · arXiv
- N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramèr, and C. Zhang. "Quantifying Memorization Across Neural Language Models". In: ICLR. 2023.
- D. Emelin, R. Le Bras, J. D. Hwang, M. Forbes, and Y. Choi. "Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences". In: EMNLP. 2021, pp. 698-718.
- J. Evertz, M. Chlosta, L. Schönherr, and T. Eisenhofer. "Whispers in the Machine: Confidentiality in LLM-integrated Systems". In: arxiv:2402.06922 (2024). · arXiv
- I. Gabriel, A. Manzini, G. Keeling, L. A. Hendricks, V. Rieser, H. Iqbal, N. Tomašev, I. Ktena, et al. "The Ethics of Advanced AI Assistants". In: arXiv:2404.16244 (2024). · arXiv
- L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. "PAL: Program- aided Language Models". In: ICML. 2023.
- D. Glukhov, I. Shumailov, Y. Gal, N. Papernot, and V. Papyan. "LLM Censorship: The Problem and its Limitations". In: International Conference on Machine Learning (2024).
- D. Hendrycks, M. Mazeika, A. Zou, S. Patel, C. Zhu, J. Navarro, D. Song, B. Li, and J. Steinhardt. "What Would Jiminy Cricket Do? Towards Agents That Behave Morally". In: NeurIPS.
- N. Kiehne, H. Kroll, and W. T. Balke. "Contextualizing language models for norms diverging from social majority". In: EMNLP. 2022.
- T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. "Large language models are zero-shot reasoners". In: NeurIPS. 2022.
- M. Komeili, K. Shuster, and J. Weston. "Internet-Augmented Dialogue Generation". In: ACL. 2022.
- N. Mireshghallah, H. Kim, X. Zhou, Y. Tsvetkov, M. Sap, R. Shokri, and Y. Choi. "Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory". In: ICLR. 2024.
- M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. "Scalable extraction of training data from (production) language models". In: arXiv:2311.17035 (2023). · arXiv
- H. Nissenbaum. "Contextual integrity up and down the data food chain". In: Theor. Inq. Law 20.1 (Mar. 2019), pp. 221-256.
- H. Nissenbaum. Privacy in context: Technology, policy, and the integrity of social life. Stanford University Press, 2009.
- A. Parisi, Y. Zhao, and N. Fiedel. "TALM: Tool Augmented Language Models". In: arXiv:2205.12255 (2022). · arXiv
- Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto. "Identifying the risks of lm agents with an lm-emulated sandbox". In: arXiv preprint arXiv:2309.15817 (2023). · arXiv
- A. Salemi, S. Mysore, M. Bendersky, and H. Zamani. "LaMP: When large language models meet personalization". In: arXiv:2304.11406 (2023). · arXiv
- N. Scherrer, C. Shi, A. Feder, and D. Blei. "Evaluating the moral beliefs encoded in llms". In: Advances in Neural Information Processing Systems 36 (2024).
- T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. "Toolformer: Language Models Can Teach Themselves to Use Tools". In: NeurIPS. 2023.
- T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. "Toolformer: Language models can teach themselves to use tools". In: NeurIPS. 2024.
- T. Schick and H. Schütze. "Generating Datasets with Pretrained Language Models". In: EMNLP.
- G. Team. Gemini: A Family of Highly Capable Multimodal Models. 2023. arXiv: 2312.11805 [cs.CL]. · arXiv
- G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. "Gemma 2: Improving Open Language Models at a Practical Size". In: arXiv preprint arXiv:2408.00118 (2024). · arXiv
- A. Trask, G. Keeling, B. Balle, S. de Haas, Y. Ibitoye, and I. Gabriel. "Privacy". In: The Ethics of Advanced AI Assistants. 2024. Chap. 13, pp. 131-137.
- E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel. "The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions". In: arXiv:2404.13208 (2024). · arXiv
- Y. Yuan, K. Tang, J. Shen, M. Zhang, and C. Wang. "Measuring Social Norms of Large Language Models". In: NAACL. 2024, pp. 650-699.
- M. Zimmer and A. Hoffman. "Privacy, Context, and Oversharing: Reputational Challenges in a Web 2.0 World". In: The Reputation Society: How Online Opinions Are Reshaping the Offline World. The MIT Press, 2012.
- A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. "Universal and transferable adversarial attacks on aligned language models". In: arXiv:2307.15043 (2023). · arXiv
Cited by
Sorted by citations
Defeating Prompt Injections by Design
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F. (2026) · 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml68715.2026.00040 · Cited by 15
Self-citation
Metadata
- Authors
- Ghalebikesabi, S., Bagdasaryan, E., Yi, R., Yona, I., Shumailov, I., Pappu, A., Shi, C., Weidinger, L., Stanforth, R., Berrada, L., Kohli, P., Huang, P.-S., & Balle, B.
- Affiliations
- Google (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2408.02373
- OpenAlex
- W4406490385
- Cited by
- 11
- References
- 2
- Citations by year
- 2026: 5 · 2025: 5 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Sociology and Political Science › Privacy, Security, and Data Protection
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- contextual integrity, privacy, AI assistants, large language models, information sharing, form filling, web forms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
