HarnessSecurity-Bench: Do Security Mechanisms Really Protect Coding Agent Harnesses?
Open accessAbstract
Coding agent harnesses mediate tool use and authorize actions, yet their security mechanisms and runtime effects remain incompletely characterized. We present HarnessSecurity, the first systematic empirical study and benchmark of open- and closed-source coding agent harnesses. First, we derive a ten-mechanism taxonomy and then assess 400 harness-mechanism cells using independent ratings by researchers and large language model (LLM) judges. We find that about half of confirmed mechanism implementations are opt-in, while closed-source harnesses exhibit substantial evidence gaps. Second, we introduce HarnessSecurity-Bench, a benchmark of 23 tasks across five attack surfaces without sacrificing legitimate task requirements. Using separate deterministic oracles to measure task utility and attack effects with security setting comparisons, we evaluate nine mechanisms across six leading harnesses: Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code, and GitHub Copilot. Under a controlled LLM baseline GLM-5.2, we conduct 2,500 trials, recording 81,155 tool calls and over 2.2 billion tokens. Enabling auto-approve increases utility and raises attack success from 29.2% to 95.6%. Network isolation and read-only mode reduce attack effects with substantial utility losses, while command allowlisting and command denylisting reduce attack effects with a small utility loss and a utility gain, respectively. Task-level cases show that restrictions on a shared capability can obstruct both legitimate and malicious operations, and that allowed tools or commands can leave unauthorized operations reachable through alternative execution paths. Harness providers should make security settings verifiable, test alternative execution paths to protected operations, and assess attack effects alongside task utility and execution costs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The protection of information in computer systems
Saltzer, J. H., & Schroeder, M. D. (1975) · Proceedings of the IEEE · doi:10.1109/proc.1975.9939 · Cited by 2,126 · see citing works
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264 · see citing works
(2024) · doi:10.52202/079017-2636 · Cited by 139 · see citing works
A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
Kang, S., An, G., & Yoo, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660771 · Cited by 128 · see citing works
Navigating the Risks: A Survey of Security and Privacy Threats in LLM-Based Agents
Gan, Y., Yang, Y., W, Z., He, P., Zeng, R., Wang, Y., Li, Q., Zhou, C., Li, S., Wang, T., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3807666 · Cited by 109 · see citing works
Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM AgentsPreprint
Cartagena, A., & Teixeira, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.16943 · Cited by 30 · see citing works
"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding EditorsPreprint
Liu, Y., Zhao, Y.-J., Lyu, Y., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2509.22040 · Cited by 19 · see citing works
RedCode: Risky Code Execution and Generation Benchmark for Code Agents
Guo, C., Liu, X., Xie, C., et al. (2024) · neural information processing systems · doi:10.52202/079017-3369 · Cited by 14 · see citing works
Rashidi, M. (2026) · arXiv.org · doi:10.48550/arxiv.2607.05743 · Cited by 4 · see citing works
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Lin, J., Liu, S., Pan, C., Lin, L., Dou, S., Xi, Z., Huang, X., Yan, H., Han, Z., Gui, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.25850 · Cited by 1 · see citing works
ClawSafety: "Safe" LLMs, Unsafe Agents
Wei, B., Zhang, Y., Pan, J., Mei, K., Wang, X., Hamm, J., Zhu, Z., & Ge, Y. (2026) · arXiv · doi:10.48550/arxiv.2604.01438 · Cited by 1 · see citing works
Self-citationHarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety
Bai, Y., Duan, J., Peng, J., Wu, X., Liu, S., Wang, S., & Chen, T. (2026) · arXiv · doi:10.48550/arxiv.2608.17597 · cited 2×
“Second, agent security studies audit permissions and violations of information flows, evaluate attacks, defenses, and authorization policies in agent workflows [10, 12, 17, 29, 57].”
Barbaste, P., Darrigol, T., Vu, G., & Wiltberger, T. (2026) · arXiv · doi:10.48550/arxiv.2609.00006 · cited 3×
“First, surveys and empirical studies classify agent defenses, harness architectures, and execution controls, with an inspection of harness source code and public product materials [2, 11, 16, 25, 27, 44].”
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Vats, N., & Golev, O. (2026) · arXiv · doi:10.48550/arxiv.2607.22585 · cited 2×
“Harness design affects task success and resource costs even when the underlying model is held fixed [18, 52].”
An Empirical Study of Harness Design for Coding Agents
Fan, R.-Z., Zhang, Z., Ma, S., Hu, Y., Wang, S., Song, K., Liu, F., Zamani, H., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2609.20804 · cited 3×
“Harness design affects task success and resource costs even when the underlying model is held fixed [18, 52].”
Explainable Fault Localization for Programming Assignments via LLM-Guided Annotation
Liu, F., Wang, T., Zhang, L., Yang, Z., Jiang, J., & Sun, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00167
QueryIPI: Query-agnostic Indirect Prompt Injection on Coding AgentsPreprint
Xie, Y., Liu, Z., Luo, M., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.23675
Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks
Schmotz, D., Beurer-Kellner, L., Abdelnabi, S., & Andriushchenko, M. (2026) · arXiv · doi:10.48550/arxiv.2602.20156
The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
Kim, J., Liu, X., Wang, Z., Qiu, S., Li, B., Guo, W., & Song, D. (2026) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.48550/arxiv.2603.11088
Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill EcosystemsPreprint
Qu, Y., Liu, Y., Geng, T., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.03081
Auditing Agent Harness SafetyPreprint
Liu, C., Guo, Y., Liu, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.14271
SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated ConstructionPreprint
Ning, Y., Zhang, Z., Lal, Y. K., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.02540
From Confident Closing to Silent Failure: Characterizing False Success in LLM AgentsPreprint
Advani, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.09863
IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue RequestsPreprint
Singh, A., Yang, J., & Chen, T.-H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.20759
SHE: Trajectory-driven Safety Harness Evolution for LLM AgentsPreprint
Qu, W., Mao, Q., Li, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.09885
Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM AgentsPreprint
Ruan, J., Li, P., Chen, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.12977
Flagged first, then by citations
Research works printed but not linked: 8
- [12][12] Dimitrios Stamatios Bouras, Yihan Dai, and Sergey Mechtaev. 2026. Authority Is Not a String: A Capability-Scoped Harness for Prompt-Injection-Resistant Coding Agents. arXiv:2609.08371 https://arxiv.org/abs/2609.08371v1 Version 1. - · arXiv
- [15][15] Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024. AgentPoison: Red-Teaming LLM Agents via Poisoning Memory or Knowledge Bases. In Advances in Neural Information Processing Systems , Vol. 37. doi: 10.52202/ 079017-4136 -
- [26][26] Daoyuan Li. 2026. RepoGuardBench: Repository-Borne Prompt Injection Attacks and Lightweight Defenses for Local Coding Agents. 5th Deep Learning for Code Workshop at ICML. https://github.com/DaoyuanLi2816/RepoGuardBench/ blob/main/paper/main.pdf Non-archival workshop presentation; official evaluation artifacts. -
- [27][27] Junjie Li, Xi Xiao, Yunbei Zhang, Chen Liu, Lin Zhao, Xiaoying Liao, Yingrui Ji, Janet Wang, Yingqiang Ge, Weijie Xu, Xi Fang, Xiang Xu, Tianchen Zhao, Youngeun Kim, Jihun Hamm, Tianyang Wang, and Chandan K. Reddy. 2026. Agent Harness Engineering: A Survey. https://picrew.github.io/LLM-Harness/main.pdf Survey preprint. -
- [50][50] Anselm L. Strauss and Juliet M. Corbin. 1998. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (2 ed.). SAGE Publications, Thousand Oaks, CA, USA. -
- [56][56] Yu Yang, Zhenyuan Li, Xiandong Ran, Jiahao Liu, Jiahui Wang, Bo Yu, and Shouling Ji. 2026. Towards Scalable and Interpretable Mobile App Risk Analysis via Large Language Models. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE) . https://arxiv.org/abs/2508.15606 - · arXiv
- [57][57] Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2025. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. In The Thirteenth International Conference on Learning Representations . https://proceedings.iclr.cc/paper\_files/ paper/2025/hash/5750f91d8fb9d5c02bd8ad2c3b44456b-Abstract-Conference.html -
- [58][58] Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2025. PoisonedRAG: Knowledge Corruption Attacks to RetrievalAugmented Generation of Large Language Models. In 34th USENIX Security Symposium (USENIX Security 25) . USENIX Association, 3827-3844. https://www.usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag
- Authors
- Zhu, Z., Huang, L., Ji, R., Ye, M., Zhou, Z., Guo, H., Wu, J., Ye, Y., Feng, Y., Dai, H.-N., & Zheng, Z.
- Affiliations
- Sun Yat-sen University (CN)
- East China Normal University (CN)
- Hong Kong Baptist University (HK)
- Tsinghua University (CN)
- Peng Cheng Laboratory (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2610.07639
- OpenAlex
- W7221035517
- Cited by
- 0
- References
- 26
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Keywords
- security mechanisms, security benchmarks, attack surface, network isolation, GitHub Copilot, Claude Code
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
