CoChat
Sign Up
PreprintCited by 2Adversarial Robustness in Machine LearningBlockchain Technology Applications and SecuritySecurity and Verification in Computing

Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agents

Open access
Patlan, Atharv Singh, Sheng, Peiyao, Hebbar, S. Ashwin, Mittal, Prateek, Viswanath, Pramod·2025·arXiv
DOI 10.48550/arxiv.2503.16248arXiv 2503.16248
Open this paper in CoChat — free accountOpen PDF

Abstract

AI agents integrated with Web3 offer autonomy and openness but raise security concerns as they interact with financial protocols and immutable smart contracts. This paper investigates the vulnerabilities of AI agents within blockchain-based financial ecosystems when exposed to adversarial threats in real-world scenarios. We introduce the concept of context manipulation -- a comprehensive attack vector that exploits unprotected context surfaces, including input channels, memory modules, and external data feeds. It expands on traditional prompt injection and reveals a more stealthy and persistent threat: memory injection. Using ElizaOS, a representative decentralized AI agent framework for automated Web3 operations, we showcase that malicious injections into prompts or historical records can trigger unauthorized asset transfers and protocol violations which could be financially devastating in reality. To quantify these risks, we introduce CrAIBench, a Web3-focused benchmark covering 150+ realistic blockchain tasks. such as token transfers, trading, bridges, and cross-chain interactions, and 500+ attack test cases using context manipulation. Our evaluation results confirm that AI models are significantly more vulnerable to memory injection compared to prompt injection. Finally, we evaluate a comprehensive defense roadmap, finding that prompt-injection defenses and detectors only provide limited protection when stored context is corrupted, whereas fine-tuning-based defenses substantially reduce attack success rates while preserving performance on single-step tasks. These results underscore the urgent need for AI agents that are both secure and fiduciarily responsible in blockchain environments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.16248

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing work typically combines outcome-level correctness with retrieval-oriented assessment and judge-based rubrics, depending on whether the benchmark exposes a memory module, uses long-context prompting, or evaluates agents acting in a specific environment, scenario, or task.

    “As a result, existing work typically combines outcome-level correctness with retrievaloriented assessment and judge-based rubrics, depending on whether the benchmark exposes a memory module, uses long-context prompting, or evaluates agents acting in a specific environment, scenario, or task (Patlan et al., 2025; Yadav et al., 2025).”

    From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Patlan et al., 2025

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. [2][2] S. Walters, S. Gao, S. Nerd, F. Da, W. Williams, T.-C. Meng, H. Han, F. He, A. Zhang, M. Wu et al. , 'Eliza: A web3 friendly ai agent operating system,' arXiv preprint arXiv:2501.06781 , 2025. - · arXiv
  2. [5][5] Y. Liu, G. Deng, Y. Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng et al. , 'Prompt injection attack against llm-integrated applications,' arXiv preprint arXiv:2306.05499 , 2023. - · arXiv
  3. [6][6] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, 'Not what you've signed up for: Compromising realworld llm-integrated applications with indirect prompt injection,' in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security , 2023, pp. 79-90. -
  4. [7][7] S. Chen, J. Piet, C. Sitawarin, and D. Wagner, 'Struq: Defending against prompt injection with structured queries,' arXiv preprint arXiv:2402.06363 , 2024. - · arXiv
  5. [8][8] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, ' τ -bench: A benchmark for tool-agent-user interaction in real-world domains,' arXiv preprint arXiv:2406.12045 , 2024. - · arXiv
  6. [9][9] A. Li, Y. Zhou, V. C. Raghuram, T. Goldstein, and M. Goldblum, 'Commercial llm agents are already vulnerable to simple yet dangerous attacks,' arXiv preprint arXiv:2502.08586 , 2025. - · arXiv
  7. [10][10] K. Hines, G. Lopez, M. Hall, F. Zarfati, Y. Zunger, and E. Kiciman, 'Defending against indirect prompt injection attacks with spotlighting,' arXiv preprint arXiv:2403.14720 , 2024. - · arXiv
  8. [11][11] Y. Liu, Y. Jia, J. Jia, D. Song, and N. Z. Gong, ' DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks ,' in 2025 IEEE Symposium on Security and Privacy (SP) . Los Alamitos, CA, USA: IEEE Computer Society, May 2025, pp. 2190-2208. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/SP61157.2025. 00250 - · DOI
  9. [16][16] D. Pasquini, M. Strohmeier, and C. Troncoso, 'Neural exec: Learning (and learning from) execution triggers for prompt injection attacks,' in Proceedings of the 2024 Workshop on Artificial Intelligence and Security , 2024, pp. 89-100. -
  10. [17][17] Q. Zhan, Z. Liang, Z. Ying, and D. Kang, 'Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents,' arXiv preprint arXiv:2403.02691 , 2024. - · arXiv
  11. [18][18] C. H. Wu, J. Y. Koh, R. Salakhutdinov, D. Fried, and A. Raghunathan, 'Adversarial attacks on multimodal agents,' arXiv preprint arXiv:2406.12814 , 2024. - · arXiv
  12. [19][19] E. Debenedetti, J. Zhang, M. Balunovi´ c, L. Beurer-Kellner, M. Fischer, and F. Tram` er, 'Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents,' arXiv preprint arXiv:2406.13352 , 2024. - · arXiv
  13. [21][21] S. Chen, A. Zharmagambetov, S. Mahloujifar, K. Chaudhuri, and C. Guo, 'Aligning llms to be robust against prompt injection,' arXiv preprint arXiv:2410.05451 , 2024. - · arXiv
  14. [22][22] E. Debenedetti, I. Shumailov, T. Fan, J. Hayes, N. Carlini, D. Fabian, C. Kern, C. Shi, A. Terzis, and F. Tram` er, 'Defeating prompt injections by design,' arXiv preprint arXiv:2503.18813 , 2025. - · arXiv
  15. [23][23] Z. Chen, M. Kang, and B. Li, 'Shieldagent: Shielding agents via verifiable safety policy reasoning,' arXiv preprint arXiv:2503.22738 , 2025. - · arXiv
  16. [24][24] F. Perez and I. Ribeiro, 'Ignore previous prompt: Attack techniques for language models,' arXiv preprint arXiv:2211.09527 , 2022. - · arXiv
  17. [25][25] Z. Xu, Y. Liu, G. Deng, Y. Li, and S. Picek, 'Llm jailbreak attack versus defense techniques-a comprehensive study,' arXiv preprint arXiv:2402.13457 , 2024. - · arXiv
  18. [26][26] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. K¨ uttler, M. Lewis, W.-t. Yih, T. Rockt¨ aschel, S. Riedel, and D. Kiela, 'Retrieval-augmented generation for knowledge-intensive nlp tasks,' in Proceedings of the 34th International Conference on Neural Information Processing Systems , ser. NIPS '20. Red Hook, NY, USA: Curran Associates Inc., 2020. -
  19. [27][27] W. Zou, R. Geng, B. Wang, and J. Jia, 'Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models,' arXiv preprint arXiv:2402.07867 , 2024. - · arXiv
  20. [28][28] Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, 'Agentpoison: Redteaming llm agents via poisoning memory or knowledge bases,' arXiv preprint arXiv:2407.12784 , 2024. - · arXiv
  21. [29][29] C. Xiang, T. Wu, Z. Zhong, D. Wagner, D. Chen, and P. Mittal, 'Certifiably robust rag against retrieval corruption,' 2024. [Online]. Available: https://arxiv.org/abs/2405.15556 - · arXiv
  22. [30][30] Z. Xiang, F. Jiang, Z. Xiong, B. Ramasubramanian, R. Poovendran, and B. Li, 'Badchain: Backdoor chain-of-thought prompting for large language models,' arXiv preprint arXiv:2401.12242 , 2024. - · arXiv
  23. [31][31] S. Dong, S. Xu, P. He, Y. Li, J. Tang, T. Liu, H. Liu, and Z. Xiang, 'A practical memory injection attack against llm agents,' arXiv preprint arXiv:2503.03704 , 2025. - · arXiv
  24. [33][33] W. Yang, X. Bi, Y. Lin, S. Chen, J. Zhou, and X. Sun, 'Watch out for your agents! investigating backdoor threats to llm-based agents,' arXiv preprint arXiv:2402.11208 , 2024. - · arXiv
  25. [35][35] S. Altman, 'Chatgpt updated memory,' Apr 2025. [Online]. Available: https://x.com/sama/status/1910380643772665873 - [53] Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto, 'Identifying the risks of lm agents with an lm-emulated sandbox,' arXiv preprint arXiv:2309.15817 , 2023. - [54] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine et al. , 'Llama guard: Llm-based input-output safeguard for human-ai conversations,' arXiv preprint arXiv:2312.06674 , 2023. - [55] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson et al. , 'Language models (mostly) know what they know,' arXiv preprint arXiv:2207.05221 , 2022. - · arXiv
  26. [37][37] etherscan.io, 'Successful Sepolia transaction with Memory Injection,' https://sepolia.etherscan.io/tx/ 0x1a2cf99a3382250f76a03b27096f2e5dfe24729bc91c63b09a3981d585b262c1, [Accessed 13-03-2025]. [57] - [56] H. Brown, L. Lin, K. Kawaguchi, and M. Shieh, 'Self-evaluation as a defense against adversarial attacks on llms,' arXiv preprint arXiv:2407.03234 , 2024. - U. Iqbal, T. Kohno, and F. Roesner, 'Llm platform security: Applying a systematic evaluation framework to openai's chatgpt plugins,' arXiv preprint arXiv:2309.10254 , 2023. - · arXiv
  27. [39][39] A. Bondarenko, D. Volk, D. Volkov, and J. Ladish, 'Demonstrating specification gaming in reasoning models,' 2025. [Online]. Available: https://arxiv.org/abs/2502.13295 - · arXiv
  28. [41][41] Z. Liu, T. Hoang, J. Zhang, M. Zhu, T. Lan, S. Kokane, J. Tan, W. Yao, Z. Liu, Y. Feng et al. , 'Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets,' arXiv preprint arXiv:2406.18518 , 2024. - · arXiv
  29. [42][42] H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu, 'Webvoyager: Building an end-to-end web agent with large multimodal models,' arXiv preprint arXiv:2401.13919 , 2024. - · arXiv
  30. [43][43] I. Nakash, G. Kour, G. Uziel, and A. Anaby-Tavor, 'Breaking react agents: Foot-in-the-door attack will get you in,' arXiv preprint arXiv:2410.16950 , 2024. - · arXiv
  31. [47][47] E. Parliament and C. of the European Union, 'Regulation (eu) 2023/xxx of the european parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts,' 2023, official Journal of the European Union. [Online]. Available: https://eur-lex.europa.eu/ legal-content/EN/TXT/?uri=CELEX%3A52021PC0206 -
  32. [50][50] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , 'The llama 3 herd of models,' arXiv preprint arXiv:2407.21783 , 2024. - · arXiv
  33. [52][52] B. Wang, C. Xu, S. Wang, Z. Gan, Y. Cheng, J. Gao, A. H. Awadallah, and B. Li, 'Adversarial glue: A multi-task benchmark for robustness evaluation of language models,' arXiv preprint arXiv:2111.02840 , 2021. - [58] E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng et al. , 'Sleeper agents: Training deceptive llms that persist through safety training,' arXiv preprint arXiv:2401.05566 , 2024. · arXiv

Cited by

Sorted by citations

Authors
Patlan, A. S., Sheng, P., Hebbar, S. A., Mittal, P., & Viswanath, P.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2503.16248
arXiv
2503.16248
OpenAlex
W6948007811
Cited by
0
References
1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Information Systems › Blockchain Technology Applications and Security
  • Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
Keywords
AI agent security, prompt injection, blockchain vulnerabilities, prompt injection defense, smart contracts

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review