CoChat
Sign Up
PreprintCited by 0Artificial Intelligence ApplicationsTopic ModelingNatural Language Processing Techniques

PostTrainBench: Can LLM Agents Automate LLM Post-Training?

Open access
Rank, Ben, Bhatnagar, Hardik, Prabhu, Ameya, Eisenberg, Shira, Nguyen, Karina, Bethge, Matthias, Andriushchenko, Maksym·2026·arXiv
DOI 10.48550/arxiv.2603.08640arXiv 2603.08640
Open this paper in CoChat — free accountOpen PDF

Abstract

AI agents have become surprisingly proficient at software engineering over the past year, largely due to improvements in reasoning capabilities. This raises a deeper question: can these systems extend their capabilities to automate AI research itself? In this paper, we explore post-training, the critical phase that turns base LLMs into useful assistants. We introduce PostTrainBench to benchmark how well LLM agents can perform post-training autonomously under bounded compute constraints (10 hours on one H100 GPU). We ask frontier agents (e.g., Claude Code with Opus 4.6) to optimize the performance of a base LLM on a particular benchmark (e.g., Qwen3-4B on AIME). Importantly, we do not provide any predefined strategies to the agents and instead give them full autonomy to find necessary information on the web, run experiments, and curate data. We find that frontier agents make substantial progress but generally lag behind instruction-tuned LLMs from leading providers: 23.2% for the best agent vs. 51.1% for official instruction-tuned models. However, agents can exceed instruction-tuned models in targeted scenarios: GPT-5.1 Codex Max achieves 89% on BFCL with Gemma-3-4B vs. 67% for the official model. We also observe several failure modes worth flagging. Agents sometimes engage in reward hacking: training on the test set, downloading existing instruction-tuned checkpoints instead of training their own, and using API keys they find to generate synthetic data without authorization. These behaviors are concerning and highlight the importance of careful sandboxing as these systems become more capable. Overall, we hope PostTrainBench will be useful for tracking progress in AI R&D automation and for studying the risks that come with it. Website and code are available at https://posttrainbench.com/.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.08640

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Injecting data exfiltration instructions into encrypted reasoning blocks from PostTrainBench long-horizon traces using Opus 4.7 in Claude Code scaffold, and letting Haiku 4.5 generate a thought about the injection, causes Opus 4.7 to follow the injected instruction and upload the file after every change when the trace is continued.

    “We ran this attack on the Haiku 4.5 and Opus 4.7 model pair.We injected data exfiltration instructions into encrypted reasoning blocks long-horizon traces from PostTrainBench (Rank et al., 2026), using Opus 4.7 in Claude Code scaffold.”

    From Stealing Reasoning Traces from Proprietary LLM APIs (2026) · cites this paper as Rank et al., 2026

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Research works printed but not linked: 31

  1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Mané, D. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. · arXiv
  2. Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero- Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Val- lone, A., Beutel, A., et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025. · arXiv
  3. Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patward- han, T., Weng, L., and Mądry, A. Mle-bench: Evaluat- ing machine learning agents on machine learning engi- neering. In International Conference on Learning Represen- tations (ICLR), 2025.
  4. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brock- man, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., Mc- Candlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
  5. Clymer, J., Duan, I., Cundy, C., Duan, Y., Heide, F., Lu, C., Mindermann, S., McGurk, C., Pan, X., Siddiqui, S., Wang, J., Yang, M., and Zhan, X. Bare minimum miti- gations for autonomous AI development. arXiv preprint arXiv:2504.15416, 2025. · arXiv
  6. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
  7. Gemma Team. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. · arXiv
  8. Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Ding, H., Gao, H., Qu, H., Li, H., Guo, J., Li, J., Chen, J., Yuan, J., Tu, J., Qiu, J., Li, J., Cai, J. L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., You, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Zhao, L., Wang, L., Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Zhou, M., Li, M., Wang, M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen, Q., Du, Q., Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R. L., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., Wu, S., Yun, T., Pei, T., Sun, T., Wang, T., Zeng, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Ma, Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z., and Zhang, Z. Deepseek-r1 incen- tivizes reasoning in llms through reinforcement learn- ing. Nature, 645(8081):633-638, September 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http: //dx.doi.org/10.1038/s41586-025-09422-z. · DOI · Paper page
  9. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv. org/abs/2106.09685.
  10. Huang, Q., Vora, J., Liang, P., and Leskovec, J. MLAgent- Bench: Evaluating language agents on machine learning experimentation. In International Conference on Machine Learning (ICML), 2024.
  11. Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., Bloom, R., Broadley, T., Du, H., Goodrich, B., Jurkovic, N., Miles, L. H., Nix, S., Lin, T., Parikh, N., Rein, D., Sato, L. J. K., Wijk, H., Ziegler, D. M., Barnes, E., and Chan, L. Measuring AI ability to complete long tasks. arXiv preprint arXiv:2503.14499, 2025. · arXiv
  12. Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and bench- builder pipeline. arXiv preprint arXiv:2406.11939, 2024a. · arXiv
  13. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The AI scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024. · arXiv
  14. Novikov, A., Vũ, N., Eisenberger, M., Dupont, E., Huang, P.-S., Wagner, A. Z., Shirobokov, S., Kozlovskii, B., Ruiz, F. J. R., Mehrabian, A., Kumar, M. P., See, A., Chaud- huri, S., Holland, G., Davies, A., Nowozin, S., Kohli, P., and Balog, M. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. · arXiv
  15. Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., and Gonzalez, J. E. The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evalua- tion of large language models. In International Conference on Machine Learning (ICML), 2025.
  16. Press, O., Amos, B., Zhao, H., Wu, Y., Ainsworth, S. K., Krupke, D., Kidger, P., Sajed, T., Stellato, B., Park, J., Bosch, N., Meril, E., Steppi, A., Zharmagambetov, A., Zhang, F., Perez-Pineiro, D., Mercurio, A., Zhan, N., Abramovich, T., Lieret, K., Zhang, H., Huang, S., Bethge, M., and Press, O. AlgoTune: Can language models speed up general-purpose numerical programs? arXiv preprint arXiv:2507.15887, 2025. · arXiv
  17. Qiang, R., Zhuang, Y., Li, Y., Sagar V K, D., Zhang, R., Li, C., Wong, I. S.-H., Yang, S., Liang, P., Zhang, C., and Dai, B. MLE-Dojo: Interactive environments for machine learning engineering. arXiv preprint arXiv:2505.07782, 2025. · arXiv
  18. Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. · arXiv
  19. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model, 2024. URL https://arxiv.org/abs/2305.18290. · arXiv
  20. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level Google-Proof Q&A benchmark. In In- ternational Conference on Learning Representations (ICLR), 2024.
  21. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseek- math: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv. org/abs/2402.03300.
  22. Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thompson, W., et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025. · arXiv
  23. Tang, J., Xia, L., Li, Z., and Huang, C. AI-Researcher: Autonomous scientific innovation. arXiv preprint arXiv:2505.18705, 2025. · arXiv
  24. Tie, G., Zhou, P., and Sun, L. A survey of AI scientists. arXiv preprint arXiv:2510.23045, 2025. · arXiv
  25. Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., Ericheva, E., Garcia, K., Goodrich, B., Jurkovic, N., Karnofsky, H., Kinniment, M., Lajko, A., Nix, S., Sato, L., Saunders, W., Taran, M., West, B., and Barnes, E. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv preprint arXiv:2411.15114, 2024. · arXiv
  26. Xu, S., Zhou, Y., Wang, W., Min, J., Yin, Z., Dai, Y., Liu, S., Pang, L., Chen, Y., and Zhang, J. Tiny model, big logic: Diversity-driven optimization elicits large-model reasoning ability in vibethinker-1.5 b. arXiv preprint arXiv:2511.06221, 2025. · arXiv
  27. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
  28. Zhang, J., Hu, S., Lu, C., Lange, R., and Clune, J. The Darwin-Gödel Machine: Open-ended evolution of self- improving agents. arXiv preprint arXiv:2505.22954, 2025. · arXiv
  29. Zhao, B., Magka, D., Jiang, M., Li, X., Raileanu, R., Shav- rina, T., Gagnon-Audet, J.-C., Niu, K., Sodhani, S., Shvartsman, M., Lupu, A., Lupidi, A., Toledo, E., Ham- bardzumyan, K., Josifoski, M., Foster, T., Cipolina-Kun, L., Charnalia, A., Dunfield, D., Miller, A. H., Mac Aodha, O., Foerster, J., and Bachrach, Y. The automated LLM speedrunning benchmark: Reproducing NanoGPT im- provements. arXiv preprint arXiv:2506.22419, 2025. · arXiv
  30. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neu- ral Information Processing Systems (NeurIPS), 2023.
  31. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. · arXiv

Authors
Rank, B., Bhatnagar, H., Prabhu, A., Eisenberg, S., Nguyen, K., Bethge, M., & Andriushchenko, M.
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.08640
arXiv
2603.08640
OpenAlex
W7134826263
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • International Max Planck Research School for Environmental, Cellular and Molecular Microbiology
  • Good Ventures Foundation
  • Bundesministerium für Bildung und Forschung
  • International Max Planck Research School for Advanced Methods in Process and Systems Engineering
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
LLM agents, post-training, benchmarking, reward hacking, sandboxing, large language models, fine-tuning, data curation, Claude Code, instruction-tuned models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review