CoChat
Sign Up
PreprintCited by 0Security and Verification in ComputingAdversarial Robustness in Machine LearningPrivacy-Preserving Technologies in Data

Permissive Information-Flow Analysis for Large Language Models

Open access
Siddiqui, Shoaib Ahmed, Gaonkar, Radhika, Köpf, Boris, Krueger, David, Paverd, Andrew, Salem, Ahmed, Tople, Shruti, Wutschitz, Lukas +2 more·2024·arXiv
DOI 10.48550/arxiv.2410.03055arXiv 2410.03055
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) are rapidly becoming commodity components of larger software systems. This poses natural security and privacy problems: poisoned data retrieved from one component can change the model's behavior and compromise the entire system, including coercing the model to spread confidential data to untrusted components. One promising approach is to tackle this problem at the system level via dynamic information flow (aka taint) tracking. Unfortunately, this approach of propagating the most restrictive input label to the output is too conservative for applications where LLMs operate on inputs retrieved from diverse sources. In this paper, we propose a novel, more permissive approach to propagate information flow labels through LLM queries. The key idea behind our approach is to propagate only the labels of the samples that were influential in generating the model output and to eliminate the labels of unnecessary inputs. We implement and investigate the effectiveness of two variations of this approach, based on (i) prompt-based retrieval augmentation, and (ii) a $k$-nearest-neighbors language model. We compare these with a baseline that uses introspection to predict the output label. Our experimental results in an LLM agent setting show that the permissive label propagator improves over the baseline in more than 85% of the cases, which underscores the practicality of our approach.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.03055

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Research works printed but not linked: 35

  1. Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. AI agents with formal security guarantees. In ICML 2024 Next Generation of AI Safety Workshop, 2024.
  2. Fu Bang. GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 212-218, 2023.
  3. Pablo Buiras, Deian Stefan, and Alejandro Russo. On dynamic flow-sensitive floating-label systems. In 27th IEEE Computer Security Foundations Symposium, CSF 2014, pages 65-79. IEEE Computer Society, 2014.
  4. James Clause, Wanchun Li, and Alessandro Orso. Dy- tan: a generic dynamic taint analysis framework. In 2007 international symposium on Software testing and analysis, pages 196-206, 2007.
  5. Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry. ContextCite: Attributing model generation to context. arXiv preprint arXiv:2409.00729, 2024. · arXiv
  6. Dorothy E Denning. A lattice model of secure informa- tion flow. Communications of the ACM, 19(5):236-243, 1976.
  7. Dominique Devriese and Frank Piessens. Noninter- ference through secure multi-execution. In 31st IEEE Symposium on Security and Privacy, SP 2010, 16-19 May 2010, Berleley/Oakland, California, USA, pages 109-124. IEEE Computer Society, 2010.
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. · arXiv
  9. William Enck, Peter Gilbert, Seungyeop Han, Vasant Tendulkar, Byung-Gon Chun, Landon P Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N Sheth. Taint- droid: an information-flow tracking system for realtime privacy monitoring on smartphones. ACM Transactions on Computer Systems (TOCS), 32(2):1-29, 2014.
  10. Sahra Ghalebikesabi, Eugene Bagdasaryan, Ren Yi, Itay Yona, Ilia Shumailov, Aneesh Pappu, Chongyang Shi, Laura Weidinger, Robert Stanforth, Leonard Berrada, Pushmeet Kohli, Po-Sen Huang, and Borja Balle. Op- erationalizing contextual integrity in privacy-conscious assistants. arXiv preprint arXiv:2408.02373 [cs.AI], 2024. · arXiv
  11. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on ma- chine learning, pages 3929-3938. PMLR, 2020.
  12. James W Hunt and Thomas G Szymanski. A fast al- gorithm for computing longest common subsequences. Communications of the ACM, 20(5):350-353, 1977.
  13. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. · arXiv
  14. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020.
  15. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
  16. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
  17. Chin-Yew Lin. ROUGE: A package for automatic eval- uation of summaries. In Text summarization branches out, pages 74-81, 2004.
  18. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Com- putational Linguistics, 12:157-173, 2024.
  19. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. Prompt injection attack against LLM-integrated applications. arXiv preprint arXiv:2306.05499, 2023. · arXiv
  20. Piotr Mardziel, Stephen Magill, Michael Hicks, and Mudhakar Srivatsa. Dynamic enforcement of knowledge-based security policies. In Proceedings IEEE CSF, pages 114-128. IEEE Computer Society, 2011.
  21. Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. Can LLMs keep a secret? testing privacy im- plications of language models via contextual integrity theory. In 12th International Conference on Learning Representations, ICLR 2024. OpenReview.net, 2024.
  22. Andrew C. Myers and Barbara Liskov. A decentralized model for information flow control. In SOSP, page 129-142. ACM, 1997.
  23. Tai Nguyen and Eric Wong. In-context example selec- tion with influences. arXiv preprint arXiv:2302.11042, 2023. · arXiv
  24. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. Automatic detection of fake news. In 27th International Conference on Compu- tational Linguistics (COLING), pages 3391-3401. ACL, 2018.
  25. Willard Rafnsson and Andrei Sabelfeld. Secure multi- execution: Fine-grained, declassification-aware, and transparent. J. Comput. Secur., 24(1):39-90, 2016.
  26. Andrei Sabelfeld and Andrew C Myers. Language- based information-flow security. IEEE Journal on se- lected areas in communications, 21(1):5-19, 2003.
  27. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Confer- ence on Machine Learning, pages 31210-31227. PMLR, 2023.
  28. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. · arXiv
  29. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36, 2024.
  30. Philipp Vogt, Florian Nentwich, Nenad Jovanovic, Engin Kirda, Christopher Kruegel, and Giovanni Vigna. Cross site scripting prevention with dynamic data tainting and static analysis. In NDSS, volume 2007, page 12, 2007.
  31. Dennis Volpano, Cynthia Irvine, and Geoffrey Smith. A sound type system for secure flow analysis. Journal of computer security, 4(2-3):167-187, 1996.
  32. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Jo- hannes Heidecke, and Alex Beutel. The instruction hier- archy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208 [cs.CR], 2024. · arXiv
  33. Lukas Wutschitz, Boris Köpf, Andrew Paverd, Sara- van Rajmohan, Ahmed Salem, Shruti Tople, Santiago Zanella-Béguelin, Menglin Xia, and Victor Rühle. Re- thinking privacy in machine learning pipelines from an information flow control perspective. arXiv preprint arXiv:2311.15792 [cs.LG], 2023. · arXiv
  34. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023. · arXiv
  35. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming lan- guage models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. · arXiv

Authors
Siddiqui, S. A., Gaonkar, R., Köpf, B., Krueger, D., Paverd, A., Salem, A., Tople, S., Wutschitz, L., Xia, M., & Zanella-Béguelin, S.
Affiliations
  • University of Cambridge (GB)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.03055
arXiv
2410.03055
OpenAlex
W4403885558
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
Keywords
large language models, information flow analysis, taint tracking, label propagation, data poisoning, privacy, LLM agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review