Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
Open accessAbstract
Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities such as skipping verification steps, inferring answers from task-adjacent metadata, or tampering with evaluation-relevant functions. RHB supports independent and chained task regimes, where chain length acts as a proxy for longer-horizon agent behavior. We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek. Exploit rates range from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero), varying sharply by post-training style. A controlled sibling comparison (DeepSeek-V3 vs. DeepSeek-R1-Zero) shows RL post-training is associated with substantially higher reward hacking (0.6% vs. 13.9%), with consistent gaps across all four task families. We identify six exploit categories and find that 72% of reward hacking episodes include explicit chain-of-thought rationale, suggesting models often frame exploits as legitimate problem-solving. Simple environmental hardening reduces exploit rates by 5.7 percentage points (87.7% relative) without degrading task success. Models with near-zero exploit rates on standard tasks show elevated rates on harder variants, suggesting that production-aligned post-training appears to suppress reward hacking only below a complexity threshold where honest solutions remain tractable.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RL post-training increases exploit rates from 0.6% to 13.9% (Thaman, 2026).
“RHB (Thaman, 2026) finds RL post-training increases exploit rates from 0.6% to 13.9%.”
From SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (2026) · cites this paper as Thaman, 2026
Table 3 in §6.2 reports overall exploit rates of 0.6% for DeepSeek-V3 and 13.9% for DeepSeek-R1-Zero. However, §6.1 says the models differ in their full post-training pipelines and treats the gap as an association with RL-dominated post-training, not a clean demonstration that RL itself caused the increase. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Research works printed but not linked: 13
- Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., and Evans, O. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms, 2025. URL https://arxiv.org/ abs/2502.17424. · arXiv
- Fletcher, L., Levoso, V., Thaman, K., and Kilianovski, M. Implicit meta-learning in small transformer mod- els: Insights from a toy task. In Proceedings of Black- boxNLP 2024, 2024. URL https://openreview. net/forum?id=rvGeznBEjw.
- Haimes, J., Wenner, C., Thaman, K., Tashev, V., Neo, C., Kran, E., and Schreiber, J. Benchmark inflation: Re- vealing llm performance gaps using retro-holdouts, 2024. URL https://arxiv.org/abs/2410.09247. · arXiv
- Jaghouar, S., Mattern, J., Ong, J. M., Straube, J., Basra, M., Pazdera, A., Thaman, K., Ferrante, M. D., Gabriel, F., Obeid, F., Erdem, K., Keiblinger, M., and Hagemann, J. INTELLECT-2: A reasoning model trained through glob- ally decentralized reinforcement learning, 2025. URL https://arxiv.org/abs/2505.07291. · arXiv
- Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues?, 2023. URL https://arxiv.org/abs/2310.06770. · arXiv
- Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Arx, S. V., Bloom, R., Broadley, T., Du, H., Goodrich, B., Jurkovic, N., Miles, L. H., Nix, S., Lin, T., Parikh, N., Rein, D., Sato, L. J. K., Wijk, H., Ziegler, D. M., Barnes, E., and Chan, L. Measuring ai ability to complete long tasks, 2025. URL https://arxiv.org/abs/ 2503.14499.
- Lecomte, V., Thaman, K., Schaeffer, R., Bashkansky, N., Chow, T., and Koyejo, S. What causes polysemanticity? An alternative origin story of mixed selectivity from in- cidental causes, 2023. URL https://arxiv.org/ abs/2312.03096. · arXiv
- MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Ku- tasov, J., Price, S., Bouscal, N., Bowman, S., Bricken, T., Cloud, A., Denison, C., Gasteiger, J., Greenblatt, R., Leike, J., Lindsey, J., Mikulik, V., Perez, E., Ro- drigues, A., Thomas, D., Webson, A., Ziegler, D., and Hubinger, E. Natural emergent misalignment from re- ward hacking in production rl, 2025. URL https: //arxiv.org/abs/2511.18397. METR. Details about metr's preliminary evalua- tion of claude 3.7 sonnet. METR Evaluations, 2025a. URL https://evaluations.metr.org/ claude-3-7-report/.
- Needham, J. et al. Evaluation awareness: Detecting when language models recognize tests, 2025. URL https: //arxiv.org/abs/2505.23836.
- Phan, L. et al. Humanity's last exam, 2025. URL https: //arxiv.org/abs/2501.14249.
- Taylor, M., Chua, J., Betley, J., Treutlein, J., and Evans, O. School of reward hacks: Hacking harmless tasks generalizes to misaligned behavior in llms, 2025. URL https://arxiv.org/abs/2508.17511. · arXiv
- Turpin, M., Arditi, A., Li, M., Benton, J., and Michael, J. Teaching models to verbalize reward hacking in chain- of-thought reasoning, 2025. URL https://arxiv. org/abs/2506.22777.
- Zhong, Z., Raghunathan, A., and Carlini, N. Impos- siblebench: Measuring llms' propensity of exploiting test cases, 2025. URL https://arxiv.org/abs/ 2510.20270.
- Authors
- Thaman, K.
- Corresponding author
- Thaman, Kunvar
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2605.02964
- OpenAlex
- W7160448415
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- reward hacking, language model agents, tool use, multi-step tasks, chain-of-thought, coding assistants
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
