CoChat
Sign Up
PreprintCited by 6Security and Verification in ComputingAccess Control and TrustAdversarial Robustness in Machine Learning

Securing AI Agents with Information-Flow Control

Open access
Costa, Manuel, Köpf, Boris, Kolluri, Aashish, Paverd, Andrew, Russinovich, Mark, Salem, Ahmed, Tople, Shruti, Wutschitz, Lukas +1 more·2025·arXiv
DOI 10.48550/arxiv.2505.23643arXiv 2505.23643
Open this paper in CoChat — free accountOpen PDF

Abstract

As AI agents become increasingly autonomous and capable, ensuring their security against vulnerabilities such as prompt injection becomes critical. This paper explores the use of information-flow control (IFC) to provide security guarantees for AI agents. We present a formal model to reason about the security and expressiveness of agent planners. Using this model, we characterize the class of properties enforceable by dynamic taint-tracking and construct a taxonomy of tasks to evaluate security and utility trade-offs of planner designs. Informed by this exploration, we present Fides, a planner that tracks confidentiality and integrity labels, deterministically enforces security policies, and introduces novel primitives for selectively hiding information. Its evaluation in AgentDojo demonstrates that this approach enables us to complete a broad range of tasks with security guarantees. A tutorial to walk readers through the the concepts introduced in the paper can be found at https://github.com/microsoft/fides

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.23643

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • CaMeL [23] and Fides [24] guard the single-session prompt-to-action path with plain dynamic IFC, whereas we add the cross-session memory dimension and non-malleability.

    “CaMeL [23] and Fides [24] guard the single-session prompt-to-action path with plain dynamic IFC, whereas we add the cross-session memory dimension and, crucially, non-malleability .”

    From Securing LLM-Agent Long-Term Memory Against Poisoning: Non-Malleable, Origin-Bound Authority with Machine-Checked Guarantees (2026) · cites this paper as [24]

    The Introduction describes FIDES using labels and policy checks to secure an agent’s tool actions. But Section 5 distinguishes FIDES from the basic dynamic-taint-tracking planner: FIDES adds selective variable introduction, inspection, and constrained decoding, so calling it “plain dynamic IFC” overstates the similarity. The cited passages describe an agent loop, but do not establish a cross-session comparison; the citing authors’ claim about what they add is not attributed to FIDES. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, and Mario Fritz. 2025. Get my drift? Catching LLM Task Drift with Activation Deltas. In IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2025. IEEE. https://arxiv.org/abs/2406.00799 · arXiv
  2. Md. Ahsan Ayub and Subhabrata Majumdar. 2024. Embedding-based Classifiers Can Detect Prompt Injection Attacks. In Conference on Applied Machine Learning in Information Security (CAMLIS 2024) (CEUR Workshop Proceedings, Vol. 3920). CEUR-WS.org, 257-268. https://ceur-ws.org/Vol-3920/paper15.pdf
  3. Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2024. AI Agents with Formal Security Guarantees. In ICML 2024 Next Generation of AI Safety Workshop. https://openreview.net/forum?id= c6jNHPksiZ
  4. Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2024. Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation. arXiv:2403.06988 [cs.LG] https://arxiv.org/abs/2403.06988 · arXiv
  5. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. StruQ: Defending Against Prompt Injection with Structured Queries. In 34th USENIX Security Symposium (USENIX Security '25). https://arxiv.org/ abs/2402.06363 To appear. · arXiv
  6. Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. 2025. SecAlign: Defending Against Prompt Injection with Preference Optimization. arXiv:2410.05451 [cs.CR] https://arxiv.org/abs/2410.05451 · arXiv
  7. Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramèr. 2025. Defeating Prompt Injections by Design. arXiv:2503.18813 [cs.CR] https://arxiv.org/abs/2503.18813 · arXiv
  8. Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. arXiv:2406.13352 [cs.CR] https://arxiv.org/abs/2406.13352 · arXiv
  9. Dorothy E Denning. 1976. A Lattice Model of Secure Information Flow. Commun. ACM 19, 5 (1976), 236-243. doi:10.1145/360051.360056 · DOI · Paper page
  10. William Enck, Peter Gilbert, Seungyeop Han, Vasant Tendulkar, Byung-Gon Chun, Landon P Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N Sheth. 2014. TaintDroid: an Information-Flow Tracking System for Realtime Privacy Monitoring on Smartphones. ACM Transactions on Computer Systems (TOCS) 32, 2 (2014), 1-29. doi:10.1145/2619091 · DOI
  11. Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang, Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. 2024. Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks. arXiv:2411.04468 [cs.AI] https://arxiv.org/abs/2411. 04468 · arXiv
  12. Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning. In 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023. 10932-10952. doi:10.18653/v1/2023.emnlp-main.674 · DOI · Paper page
  13. Joseph A Goguen and José Meseguer. 1982. Security Policies and Security Models. In 1982 IEEE Symposium on Security and Privacy. IEEE, 11-11. doi:10.1109/SP.1982.10014 · DOI · Paper page
  14. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173 [cs.CR] https://arxiv.org/abs/2302.12173 · arXiv
  15. Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. 2024. Defending Against Indirect Prompt Injection Attacks With Spotlighting. In Conference on Applied Machine Learning in Information Security (CAMLIS 2024) (CEUR Workshop Proceedings, Vol. 3920). CEUR-WS.org, 48-62. https://ceur-ws.org/Vol-3920/paper03.pdf
  16. Feiran Jia, Tong Wu, Xin Qin, and Anna Squicciarini. 2024. The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents. arXiv:2412.16682 [cs.CR] https://arxiv.org/ abs/2412.16682 · arXiv
  17. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023. Prompt Injection Attack against LLM-Integrated Applications. arXiv:2306.05499 [cs.CR] https://arxiv.org/abs/2306.05499 · arXiv
  18. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and Benchmarking Prompt Injection Attacks and Defenses. In 33rd USENIX Security Symposium (USENIX Security 24). USENIX Association, 1831-1847. https://www.usenix.org/conference/usenixsecurity24/presentation/ liu-yupei
  19. Andrew C. Myers and Barbara Liskov. 1997. A Decentralized Model for Information Flow Control. In 16th ACM Symposium on Operating Systems Principles, SOSP '97. ACM, 129-142. doi:10.1145/268998.266669 · DOI · Paper page
  20. Andrew C Myers, Andrei Sabelfeld, and Steve Zdancewic. 2004. Enforcing Robust Declassification. In 17th IEEE Computer Security Foundations Workshop, CSF '2004. IEEE, 172-186. doi:10.1109/CSFW.2004.1310740 · DOI
  21. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Systems, Vol.
  22. Curran Associates, Inc., 27730-27744. https://proceedings.neurips.cc/paper_files/paper/ 2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf
  23. Andrei Sabelfeld and Andrew C Myers. 2003. Language-Based Information-Flow Security. IEEE J. on Selected Areas in Communications 21, 1 (2003), 5-19. doi:10.1109/JSAC.2002.806121 · DOI · Paper page
  24. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In Advances in Neural Information Processing Systems, Vol. 36. Curran As- sociates, Inc., 68539-68551. https://proceedings.neurips.cc/paper_files/paper/2023/file/ d842425e4bf79ba039352da0f658a906-Paper-Conference.pdf
  25. Daniel Schoepe, Musard Balliu, Benjamin C. Pierce, and Andrei Sabelfeld. 2016. Explicit Secrecy: A Policy for Taint Tracking. In 2016 IEEE European Symposium on Security and Privacy. IEEE, 15-30. doi:10.1109/ EuroSP.2016.14 · DOI
  26. Shoaib Ahmed Siddiqui, Radhika Gaonkar, Boris Köpf, David Krueger, Andrew Paverd, Ahmed Salem, Shruti Tople, Lukas Wutschitz, Menglin Xia, and Santiago Zanella-Béguelin. 2024. Permissive Information-Flow Analysis for Large Language Models. arXiv:2410.03055 [cs.LG] https://arxiv.org/abs/2410.03055 · arXiv
  27. Dennis Volpano. 1999. Safety versus Secrecy. In Static Analysis (SAS 1999) (Lecture Notes in Computer Science, Vol. 1694). Springer, 303-311. doi:10.1007/3-540-48294-6_20 · DOI
  28. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. 2024. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv:2404.13208 [cs.CR] https://arxiv. org/abs/2404.13208 · arXiv
  29. Fangzhou Wu, Ethan Cecchetti, and Chaowei Xiao. 2024. System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective. arXiv:2409.19091 [cs.CR] https://arxiv.org/abs/ 2409.19091 · arXiv
  30. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2024. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversations. In First Conference on Language Modeling, COLM 2024. https://openreview.net/forum?id=BAakY1hNKS
  31. Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqiang Zhao, Ravi Agrawal, Sathish Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou. 2025. Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy. In 13th International Conference on Learning Representations. https: //openreview.net/forum?id=sjWG7B8dvt
  32. Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Gong. 2024. GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis. In 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL, 507-518. doi:10.18653/v1/2024.acl-long.30 · DOI
  33. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In 11th International Conference on Learning Representations. https://openreview.net/forum?id=WE_vluYUL-X
  34. Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models. arXiv:2312.14197 [cs.CL] https://arxiv.org/abs/2312.14197 · arXiv
  35. Peter Yong Zhong, Siyuan Chen, Ruiqi Wang, McKenna McCall, Ben L. Titzer, Heather Miller, and Phillip B. Gibbons. 2025. RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage. arXiv:2502.08966 [cs.CR] https://arxiv.org/abs/2502.08966 · arXiv
  36. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. Improving Alignment and Robust- ness with Circuit Breakers. In Advances in Neural Information Processing Systems, Vol. 37. Curran As- sociates, Inc., 83345-83373. https://proceedings.neurips.cc/paper_files/paper/2024/file/ 97ca7168c2c333df5ea61ece3b3276e1-Paper-Conference.pdf Tool content: "[ {'body': '#read_channel_messages-result-0-0.body#', 'recipient: '#read_channel_messages-result-0-0.recipient#', 'sender': '#read_channel_messages-result-0-0.sender#' }, {'body': '#read_channel_messages-result-0-1.body#', 'recipient': '#read_channel_messages-result-0-1.recipient#', 'sender': '#read_channel_messages-result-0-1.sender#' }, {'body': '#read_channel_messages-result-0-2.body#', 'recipient: '#read_channel_messages-result-0-2.recipient#', 'sender': '#read_channel_messages-result-0-2.sender#' } ]" tool_call_id: 'read_channel_messages_0' Tool content: "[ {'body': '#read_channel_messages-result-0-3.body#', 'recipient': '#read_channel_messages-result-0-3.recipient#', 'sender': '#read_channel_messages-result-0-3.sender#' } ]" tool_call_id: 'read_channel_messages_1' Tool content: "[ {'body': '#read_channel_messages-result-0-4.body#', 'recipient': '#read_channel_messages-result-0-4.recipient#', 'sender': '#read_channel_messages-result-0-4.sender#' } ]" tool_call_id: 'read_channel_messages_2' Tool content: "[]" tool_call_id: 'read_channel_messages_3' Assistant content: null tool_calls: [{ 'name': 'add_user_to_channel', 'tool_call_id': 'add_user_to_channel_0', 'arguments': { 'channel': '#get_channels-result-0-3#', 'user': 'Alice' } }] Tool content: null tool_call_id: 'add_user_to_channel_0'

Cited by

Sorted by citations

Authors
Costa, M., Köpf, B., Kolluri, A., Paverd, A., Russinovich, M., Salem, A., Tople, S., Wutschitz, L., & Zanella-Béguelin, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.23643
arXiv
2505.23643
OpenAlex
W4416712242
Cited by
6
References
0
Citations by year
2026: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Social Sciences › Sociology and Political Science › Access Control and Trust
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
AI agent security, information flow control, prompt injection, fides, formal security model

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review