Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks
Open accessAbstract
LLM agents are evolving rapidly, powered by code execution, tools, and the recently introduced agent skills feature. Skills allow users to extend LLM applications with specialized third-party code, knowledge, and instructions. Although this can extend agent capabilities to new domains, it creates an increasingly complex agent supply chain, offering new surfaces for prompt injection attacks. We identify skill-based prompt injection as a significant threat and introduce SkillInject, a benchmark evaluating the susceptibility of widely-used LLM agents to injections through skill files. SkillInject contains 202 injection-task pairs with attacks ranging from obviously malicious injections to subtle, context-dependent attacks hidden in otherwise legitimate instructions. We evaluate frontier LLMs on SkillInject, measuring both security in terms of harmful instruction avoidance and utility in terms of legitimate instruction compliance. Our results show that today's agents are highly vulnerable with up to 80% attack success rate with frontier models, often executing extremely harmful instructions including data exfiltration, destructive action, and ransomware-like behavior. They furthermore suggest that this problem will not be solved through model scaling or simple input filtering, but that robust agent security will require context-aware authorization frameworks. Our benchmark is available at https://www.skill-inject.com/.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Querying o4-mini with a prompt to internalize a backup method, then porting the reasoning block to GPT 5.6 Sol with an unrelated query, causes GPT 5.6 Sol to treat the injected trace as its own prior reasoning and produce a script that uploads the presentation to the attacker's server.
“we query o4-mini with”
From Stealing Reasoning Traces from Proprietary LLM APIs (2026) · cites this paper as Schmotz et al., 2026
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Research works printed but not linked: 26
- Abdelnabi, S., Fay, A., Salem, A., Zverev, E., Liao, K.- C., Liu, C.-H., Kuo, C.-C., Weigend, J., Manlangit, D., Apostolov, A., et al. LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge. arXiv preprint arXiv:2506.09956, 2025b. · arXiv
- Barth, A., Datta, A., Mitchell, J. C., and Nissenbaum, H. Privacy and contextual integrity: Framework and applica- tions. In IEEE symposium on security and privacy (S&P), 2006.
- Beurer-Kellner, L., Buesser, B., Cret ¸u, A.-M., Debenedetti, E., Dobos, D., Fabian, D., Fischer, M., Froelicher, D., Grosse, K., Naeff, D., et al. Design Patterns for Securing LLM Agents against Prompt Injections. arXiv preprint arXiv:2506.08837, 2025. · arXiv
- Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F. Poisoning web-scale training datasets is practical. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 407-425. IEEE, 2024.
- Chen, S., Zharmagambetov, A., Wagner, D., and Guo, C. Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks. arXiv preprint arXiv:2507.02735, 2025. · arXiv
- Chen, Z., Xiang, Z., Xiao, C., Song, D., and Li, B. Agent- poison: Red-teaming llm agents via poisoning memory or knowledge bases. Advances in Neural Information Processing Systems, 37:130185-130213, 2024.
- Costa, M., Köpf, B., Kolluri, A., Paverd, A., Russinovich, M., Salem, A., Tople, S., Wutschitz, L., and Zanella- Béguelin, S. Securing ai agents with information-flow control. arXiv preprint arXiv:2505.23643, 2025. · arXiv
- Debenedetti, E., Rando, J., Paleka, D., Florin, S., Albastroiu, D., Cohen, N., Lemberg, Y., Ghosh, R., Wen, R., Salem, A., et al. Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition. Advances in Neural Information Processing Systems, 37:36914- 36937, 2024a.
- Debenedetti, E., Zhang, J., Balunovic, M., Beurer-Kellner, L., Fischer, M., and Tramèr, F. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. In NeurIPS Datasets and Benchmarks Track, 2024b.
- Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., and Tramèr, F. Defeating Prompt Injections by Design. arXiv preprint arXiv:2503.18813, 2025. · arXiv
- Evtimov, I., Zharmagambetov, A., Grattafiori, A., Guo, C., and Chaudhuri, K. Wasp: Benchmarking web agent security against prompt injection attacks. arXiv, 2025.
- Ghalebikesabi, S., Bagdasaryan, E., Yi, R., Yona, I., Shu- mailov, I., Pappu, A., Shi, C., Weidinger, L., Stanforth, R., Berrada, L., et al. Operationalizing Contextual In- tegrity in Privacy-Conscious Assistants. Transactions on Machine Learning Research (TMLR), 2025.
- Gomaa, A., Salem, A., and Abdelnabi, S. ConVerse: Bench- marking Contextual Safety in Agent-to-Agent Conversa- tions. arXiv preprint arXiv:2511.05359, 2025. · arXiv
- Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. Not what you've signed up for: Compro- mising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In the 16th ACM workshop on Artificial Intelligence and Security, 2023.
- Hines, K., Lopez, G., Hall, M., Zarfati, F., Zunger, Y., and Kiciman, E. Defending Against Indirect Prompt Injection Attacks With Spotlighting. arXiv, 2024.
- Kuntz, T., Duzan, A., Zhao, H., Croce, F., Kolter, Z., Flam- marion, N., and Andriushchenko, M. OS-Harm: A Bench- mark for Measuring Safety of Computer Use Agents. arXiv preprint arXiv:2506.14866, 2025. · arXiv
- Liu, Y., Jia, Y., Geng, R., Jia, J., and Gong, N. Z. Formal- izing and Benchmarking Prompt Injection Attacks and Defenses. In USENIX Security, 2024.
- O'Reilly, J. eating lobster souls part ii: the supply chain (aka -backdooring the #1 downloaded clawdhub skill). http s://x.com/theonejvo/status/201589298 0851474595, 2025. X (formerly Twitter). Accessed: 2026-01-29.
- Perez, F. and Ribeiro, I. Ignore Previous Prompt: Attack Techniques For Language Models. In ML Safety NeurIPS 2022 Workshop, 2022.
- Schmotz, D., Abdelnabi, S., and Andriushchenko, M. Agent skills enable a new class of realistic and trivially sim- ple prompt injections. arXiv preprint arXiv:2510.26328, 2025. · arXiv
- Shao, Y., Li, T., Shi, W., Liu, Y., and Yang, D. Privacylens: Evaluating privacy norm awareness of language models in action. NeurIPS Datasets and Benchmarks Track, 2024.
- Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., and Beutel, A. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024. · arXiv
- Xiong, J., Zhu, C., Lin, S., Zhang, C., Zhang, Y., Liu, Y., and Li, L. Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Models. arXiv preprint arXiv:2505.16957, 2025. · arXiv
- Yi, R., Suciu, O., Gascon, A., Meiklejohn, S., Bagdasarian, E., and Gruteser, M. Privacy Reasoning in Ambiguous Contexts. In NeurIPS, 2025.
- Zou, W., Geng, R., Wang, B., and Jia, J. {PoisonedRAG}: Knowledge corruption attacks to {Retrieval-Augmented} generation of large language models. In 34th USENIX Security Symposium (USENIX Security 25), pp. 3827- 3844, 2025.
- Zverev, E., Abdelnabi, S., Tabesh, S., Fritz, M., and Lam- pert, C. H. Can llms separate instructions from data? and what do we even mean by that? arXiv preprint arXiv:2403.06833, 2024. · arXiv
- Authors
- Schmotz, D., Beurer-Kellner, L., Abdelnabi, S., & Andriushchenko, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2602.20156
- OpenAlex
- W7131420072
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Keywords
- LLM agents, prompt injection, agent security, data exfiltration, program execution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
