CoChat
Sign Up
PreprintCited by 1Software Engineering ResearchAI-based Problem Solving and PlanningSoftware Testing and Debugging Techniques

Scaling Test-Time Compute for Agentic Coding

Open access
Kim, Joongwon, Yang, Wannan, Niu, Kelvin, Zhang, Hongming, Zhu, Yun, Helenowski, Eryk, Silva, Ruan, Chen, Zhengxing +8 more·2026·arXiv
DOI 10.48550/arxiv.2604.16529arXiv 2604.16529
Open this paper in CoChat — free accountOpen PDF

Abstract

Test-time scaling has become a powerful way to improve large language models. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge is no longer generating more attempts, but representing prior experience in a form that can be effectively selected from and reused. We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories. Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes while discarding low-signal trace details. This representation enables two complementary forms of inference-time scaling. For parallel scaling, we introduce Recursive Tournament Voting (RTV), which recursively narrows a population of rollout summaries through small-group comparisons. For sequential scaling, we adapt Parallel-Distill-Refine (PDR) to the agentic setting by conditioning new rollouts on summaries distilled from prior attempts. Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0. For example, by using our method Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified (mini-SWE-agent) and 46.9% to 59.1% on Terminal-Bench v2.0 (Terminus 1). Our results suggest that test-time scaling for long-horizon agents is fundamentally a problem of representation, selection, and reuse.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2604.16529

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Kim et al. extend inference-time scaling to long-horizon coding agents.

    “Kim et al. (Kim et al., 2026) extend inference-time scaling to long-horizon coding agents.”

    From GitSwarm: Decentralized Compounding Inference (2026) · cites this paper as Kim et al., 2026

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 24

  1. Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. Otter: Generating tests from issues to validate swe patches. arXiv preprint arXiv:2502.05368, 2025. · arXiv
  2. Antonis Antoniades, Albert Örwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and William Wang. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint arXiv:2410.20285, 2024. · arXiv
  3. Yifeng Ding and Lingming Zhang. Swe-replay: Efficient test-time scaling for software engineering agents. CoRR, abs/2601.22129, 2026. doi: 10.48550/ARXIV.2601.22129. https://doi.org/10.48550/arXiv.2601.22129. · DOI
  4. Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirhoseini. Codemonkeys: Scaling test-time compute for software engineering. arXiv preprint arXiv:2501.14723, 2025. · arXiv
  5. Tony Feng, Trieu H Trinh, Garrett Bingham, Dawsen Hwang, Yuri Chervonyi, Junehyuk Jung, Joonkyung Lee, Carlo Pagano, Sang-hyun Kim, Federico Pasqualotto, et al. Towards autonomous mathematics research. arXiv preprint arXiv:2602.10177, 2026. · arXiv
  6. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, 2024. https://openreview.net/forum?id=VTF8yNQM66.
  7. Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. CoRR, abs/2502.18581, 2025. doi: 10.48550/ARXIV.2502.18581. https://doi.org/10.48550/arXiv.2502.18581. · DOI
  8. Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E. Gonzalez, and Ion Stoica. S*: Test time scaling for code generation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 15964-15978. Association for Computational Linguistics, 2025. https://aclanthology.org/2025.findings-emnlp.865/.
  9. Xiaochuan Li, Weixi Feng, Meng Cao, Reno Kriz, Bhuwan Dhingra, Yixin Nie, Emma Strubell, William Yang Wang, and Daniel Fried. Benchmark test-time scaling of general LLM agents. CoRR, abs/2602.18998, 2026. doi: 10.48550/ARXIV.2602.18998. https://doi.org/10.48550/arXiv.2602.18998. · DOI
  10. Tobias Lindenbauer, Igor Slinko, Ludwig Felder, Egor Bogomolov, and Yaroslav Zharov. The complexity trap: Simple observation masking is as efficient as LLM summarization for agent context management. CoRR, abs/2508.21433, 2025. doi: 10.48550/ARXIV.2508.21433. https://doi.org/10.48550/arXiv.2508.21433. · DOI · Paper page
  11. Miao Lu, Weiwei Sun, Weihua Du, Zhan Ling, Xuesong Yao, Kang Liu, and Jiecao Chen. Scaling LLM multi-turn RL with end-to-end summarization-based context management. CoRR, abs/2510.06727, 2025. doi: 10.48550/ARXIV. 2510.06727. https://doi.org/10.48550/arXiv.2510.06727. · DOI
  12. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023. http://papers.nips.cc/paper_files/paper/2023/hash/ 91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html.
  13. Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Ruslan Salakhutdinov, Manzil Zaheer, Sanjeev Arora, and Anirudh Goyal. Rethinking thinking tokens: Llms as improvement operators. CoRR, abs/2510.01123, 2025. doi: 10.48550/ARXIV.2510.01123. https://doi.org/10.48550/arXiv.2510.01123. · DOI · Paper page
  14. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. CoRR, abs/2601.11868, 2026. doi: 10.48550/ARXIV.2601.11868. https://doi.org/10.48550/arXiv.2601.11868. · DOI · Paper page
  15. Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tom- czak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neu- ral Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 -15, 2024, 2024. http: //papers.nips.cc/paper_files/paper/2024/hash/639d992f819c2b40387d4d5170b8ffd7-Abstract-Conference.html.
  16. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. doi: 10.48550/ARXIV.2408.03314. https://doi.org/10.48550/arXiv.2408.03314. · DOI · Paper page
  17. Siddarth Venkatraman, Vineet Jain, Sarthak Mittal, Vedant Shah, Johan S. Obando-Ceron, Yoshua Bengio, Brian R. Bartoldson, Bhavya Kailkhura, Guillaume Lajoie, Glen Berseth, Nikolay Malkin, and Moksh Jain. Recursive self-aggregation unlocks deep thinking in large language models. CoRR, abs/2509.26626, 2025. doi: 10.48550/ ARXIV.2509.26626. https://doi.org/10.48550/arXiv.2509.26626. · DOI · Paper page
  18. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. https://openreview.net/forum?id=1PL1NIMMrw.
  19. Yuan-An Xiao, Pengfei Gao, Chao Peng, and Yingfei Xiong. Improving the efficiency of LLM agent systems through trajectory reduction. CoRR, abs/2509.23586, 2025. doi: 10.48550/ARXIV.2509.23586. https://doi.org/10.48550/ arXiv.2509.23586. · DOI · Paper page
  20. Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? A survey on test-time scaling in large language models. CoRR, abs/2503.24235, 2025. doi: 10.48550/ARXIV.2503.24235. https://doi.org/10.48550/arXiv.2503.24235. · DOI
  21. Wenting Zhao, Pranjal Aggarwal, Swarnadeep Saha, Asli Celikyilmaz, Jason Weston, and Ilia Kulikov. The majority is not always right: RL training for solution aggregation. CoRR, abs/2509.06870, 2025. doi: 10.48550/ARXIV.2509. 06870. https://doi.org/10.48550/arXiv.2509.06870. · DOI
  22. King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, and Wangchunshu Zhou. Scaling test-time compute for LLM agents. CoRR, abs/2506.12928, 2025. doi: 10.48550/ARXIV.2506.12928. https://doi.org/10.48550/arXiv.2506.12928. · DOI
  23. Yufan Zhuang, Chandan Singh, Liyuan Liu, Yelong Shen, Dinghuai Zhang, Jingbo Shang, Jianfeng Gao, and Weizhu Chen. Test-time recursive thinking: Self-improvement without external feedback. CoRR, abs/2602.03094, 2026. doi: 10.48550/ARXIV.2602.03094. https://doi.org/10.48550/arXiv.2602.03094. · DOI
  24. THOUGHT: I need to install the missing dependencies first. Let me install the required packages. """bash pip install asgiref pytz sqlparse -q && python /testbed/test_fix.py """ "Script must run headlessly: 'vim -Nu NONE -n -Es /app/input.csv -S /app/apply_macros.vim'" After this command executes, I will verify it hits exit code 0, and then I will double check 'cmp' again on the full files. Once that passes, the task is 100% complete! Let's estimate the time. 10k rows took 1.2s, so 1M rows should take roughly 120s. I should set the 'timeout_sec' to at least 240s to be safe. Bash commands: '''bash cd /app vim -Nu NONE -n -Es /app/input.csv -S /app/apply_macros.vim cmp /app/input.csv /app/expected.csv echo $? ''' Let's quickly review the constraints and success criteria. -"Transform /app/input.csv (1 million rows) to match /app/expected.csv exactly using keystroke-efficient Vim macros. Save your script as /app/apply_macros.vim." -> Done, script is saved, keystroke limit is fine.

Cited by

Sorted by citations

Authors
Kim, J., Yang, W., Niu, K., Zhang, H., Zhu, Y., Helenowski, E., Silva, R., Chen, Z., Iyer, S., Zaheer, M., Fried, D., Hajishirzi, H., Arora, S., Synnaeve, G., Salakhutdinov, R., & Goyal, A.
Affiliations
  • New York University (US)
  • Google DeepMind (United Kingdom) (GB)
  • Carnegie Mellon University (US)
  • University of Washington (US)
  • Princeton University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2604.16529
arXiv
2604.16529
OpenAlex
W7155052653
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
test-time scaling, agentic coding, coding agents, long-horizon agents, large language models, SWE-bench

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review