Executable Code Actions Elicit Better LLM Agents
Open accessAbstract
Large Language Model (LLM) agents, capable of performing a broad range of actions, such as invoking tools and controlling robots, show great potential in tackling real-world challenges. LLM agents are typically prompted to produce actions by generating JSON or text in a pre-defined format, which is usually limited by constrained action space (e.g., the scope of pre-defined tools) and restricted flexibility (e.g., inability to compose multiple tools). This work proposes to use executable Python code to consolidate LLM agents' actions into a unified action space (CodeAct). Integrated with a Python interpreter, CodeAct can execute code actions and dynamically revise prior actions or emit new actions upon new observations through multi-turn interactions. Our extensive analysis of 17 LLMs on API-Bank and a newly curated benchmark shows that CodeAct outperforms widely used alternatives (up to 20% higher success rate). The encouraging performance of CodeAct motivates us to build an open-source LLM agent that interacts with environments by executing interpretable code and collaborates with users using natural language. To this end, we collect an instruction-tuning dataset CodeActInstruct that consists of 7k multi-turn interactions using CodeAct. We show that it can be used with existing data to improve models in agent-oriented tasks without compromising their general capability. CodeActAgent, finetuned from Llama2 and Mistral, is integrated with Python interpreter and uniquely tailored to perform sophisticated tasks (e.g., model training) using existing libraries and autonomously self-debug.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Interaction and code generation are increasingly used together, with code as the modality of choice for actions, tool construction, and reasoning.
“Interaction and code generation are increasingly used together, with code as the modality of choice for actions [48, 59], tool construction [13, 51, 69], and reasoning [39, 66, 67].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [48]
These agents typically operate in an iterative loop, at each turn they reason about the current state, devise a plan, and execute a tool.
“These agents, powered by LLMs, typically operate in an iterative loop [39, 29], at each turn they reason about the current state, devise a plan, and execute a tool (e.g., read a file, run tests).”
From The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management (2025) · cites this paper as [29]
Recent advances using code generation as an action space have opened new opportunities for LLM-driven automated workflows.
“Recent advances using code generation as an action space have opened new opportunities for LLM-driven automated workflows (Wang et al., 2024).”
From The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search (2025) · cites this paper as Wang et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 62
- Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N., Julian, R., Kalashnikov, D., Kuang, Y., Lee, K.-H., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor, P., Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D., Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Xu, S., Yan, M., and Zeng, A. Do as i can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691, 2022. · arXiv
- Boerner, T. J., Deems, S., Furlani, T. R., Knuth, S. L., and Towns, J. Access: Advancing innovation: Nsf's ad- vanced cyberinfrastructure coordination ecosystem: Ser- vices & support. In Practice and Experience in Advanced Research Computing, pp. 173-176. 2023.
- Bran, A. M., Cox, S., White, A. D., and Schwaller, P. Chem- crow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023. · arXiv
- Chen, B., Shu, C., Shareghi, E., Collier, N., Narasimhan, K., and Yao, S. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023a. · arXiv
- Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
- Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023b. · arXiv
- Chen, Y., Sikka, K., Cogswell, M., Ji, H., and Divakaran, A. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. arXiv preprint arXiv:2311.10081, 2023c. · arXiv
- Chen, Y., Sikka, K., Cogswell, M., Ji, H., and Di- vakaran, A. Measuring and improving chain-of-thought reasoning in vision-language models. arXiv preprint arXiv:2309.04461, 2023d. · arXiv
- Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. · arXiv
- Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- Eloundou, T., Manning, S., Mishkin, P., and Rock, D. Gpts are gpts: An early look at the labor market im- pact potential of large language models. arXiv preprint arXiv:2303.10130, 2023. · arXiv
- Fischer, K. A. Reflective linguistic programming (rlp): A stepping stone in socially-aware agi (socialagi). arXiv preprint arXiv:2305.12647, 2023. · arXiv
- Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided lan- guage models. In International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023.
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
- Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021a.
- Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
- Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al. Metagpt: Meta programming for multi-agent collabora- tive framework. arXiv preprint arXiv:2308.00352, 2023. · arXiv
- Huang, W., Wang, C., Zhang, R., Li, Y., Wu, J., and Fei-Fei, L. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023. · arXiv
- Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. · arXiv
- Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491, 2023. · arXiv
- Lee, M., Liang, P., and Yang, Q. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI conference on human factors in computing systems, pp. 1-19, 2022.
- Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07753, 2022. · arXiv
- Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023a.
- Liu, R., Yang, R., Jia, C., Zhang, G., Zhou, D., Dai, A. M., Yang, D., and Vosoughi, S. Training socially aligned lan- guage models in simulated human society. arXiv preprint arXiv:2305.16960, 2023b. · arXiv
- Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. Eureka: Human-level reward design via coding large lan- guage models. arXiv preprint arXiv:2310.12931, 2023. · arXiv
- Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pa- sunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi- Yu, J., Celikyilmaz, A., et al. Augmented language mod- els: a survey. arXiv preprint arXiv:2302.07842, 2023. · arXiv
- Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with hu- man feedback. arXiv preprint arXiv:2112.09332, 2021. · arXiv
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
- Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive sim- ulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
- Pasupat, P. and Liang, P. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470-1480, 2015.
- Patil, S. G., Zhang, T., Wang, X., and Gonza- lez, J. E. Gorilla: Large language model con- nected with massive apis. ArXiv, abs/2305.15334, 2023. URL https://api.semanticscholar. org/CorpusID:258865184. · arXiv
- Qian, C., Cong, X., Yang, C., Chen, W., Su, Y., Xu, J., Liu, Z., and Sun, M. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023. · arXiv
- Qiao, B., Li, L., Zhang, X., He, S., Kang, Y., Zhang, C., Yang, F., Dong, H., Zhang, J., Wang, L., et al. Taskweaver: A code-first agent framework. arXiv preprint arXiv:2311.17541, 2023. · arXiv
- Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023a. · arXiv
- Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y.-T., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Tian, R., Xie, R., Zhou, J., Gerstein, M. H., Li, D., Liu, Z., and Sun, M. Toolllm: Facilitating large language models to master 16000+ real-world apis. ArXiv, abs/2307.16789, 2023b. URL https://api.semanticscholar. org/CorpusID:260334759. · arXiv
- Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Tool- former: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. · arXiv
- Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580, 2023. · arXiv
- Shridhar, M., Yuan, X., Cote, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. Alfworld: Aligning text and embod- ied environments for interactive learning. In International Conference on Learning Representations, 2020.
- Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A. Progprompt: Generating situated robot task plans us- ing large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-11530, 2023. doi: 10.1109/ICRA48891.2023. 10161317. · DOI · Paper page
- Surís, D., Menon, S., and Vondrick, C. Vipergpt: Visual in- ference via python execution for reasoning. Proceedings of IEEE International Conference on Computer Vision (ICCV), 2023.
- Tang, X., Jin, Q., Zhu, K., Yuan, T., Zhang, Y., Zhou, W., Qu, M., Zhao, Y., Tang, J., Zhang, Z., et al. Prioritizing safeguarding over autonomy: Risks of llm agents for science. arXiv preprint arXiv:2402.04247, 2024. · arXiv
- Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
- Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open- ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a. · arXiv
- Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023b. · arXiv
- Wang, R., Jansen, P. A., Côté, M.-A., and Am- manabrolu, P. Scienceworld: Is your agent smarter than a 5th grader? In Conference on Empirical Methods in Natural Language Processing, 2022a. URL https://api.semanticscholar. org/CorpusID:247451124.
- Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022b. · arXiv
- Wang, X., Li, S., and Ji, H. Code4Struct: Code generation for few-shot event structure prediction. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3640-3663, Toronto, Canada, July 2023c. Association for Computational Linguistics. doi: 10.18653/v1/2023. acl-long.202. URL https://aclanthology.org/ 2023.acl-long.202. · DOI
- Wang, X., Peng, H., Jabbarvand, R., and Ji, H. Leti: Learning to generate from textual interactions. ArXiv, abs/2305.10314, 2023d. · arXiv
- Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., and Ji, H. Mint: Evaluating llms in multi-turn interac- tion with tools and language feedback. arXiv preprint arXiv:2309.10691, 2023e. · arXiv
- Wang, Z., Cai, S., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023f. · arXiv
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824-24837, 2022.
- Yang, J., Prabhakar, A., Narasimhan, K., and Yao, S. In- tercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024a.
- Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhut- dinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369- 2380, 2018.
- Yao, S., Chen, H., Yang, J., and Narasimhan, K. Webshop: Towards scalable real-world web inter- action with grounded language agents. Advances in Neural Information Processing Systems, 35:20744- 20757, 2022a.
- Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022b.
- Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate prob- lem solving with large language models. arXiv preprint arXiv:2305.10601, 2023a. · arXiv
- Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., Murthy, R., Chen, Z., Zhang, J., Arpit, D., et al. Retro- former: Retrospective large language agents with policy gradient optimization. arXiv preprint arXiv:2308.02151, 2023b. · arXiv
- Yuan, L., Chen, Y., Wang, X., Fung, Y. R., Peng, H., and Ji, H. Craft: Customizing llms by creating and retriev- ing from specialized toolsets. ArXiv, abs/2309.17428, 2023. URL https://api.semanticscholar. org/CorpusID:263310662. · arXiv
- Zhang, C., Liu, L., Wang, J., Wang, C., Sun, X., Wang, H., and Cai, M. Prefer: Prompt ensemble learning via feedback-reflect-refine. arXiv preprint arXiv:2308.12033, 2023. · arXiv
- Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. · arXiv
- Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeinterpreter: Inte- grating code generation with execution and refinement. https://arxiv.org/abs/2402.14658, 2024. · arXiv
- Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., et al. Ghost in the minecraft: Generally capable agents for open-world envi- roments via large language models with text-based knowl- edge and memory. arXiv preprint arXiv:2305.17144, 2023. · arXiv
Cited by
Sorted by citations
Generative AI without guardrails can harm learning: Evidence from high school mathematics
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2422633122 · Cited by 309 · see citing works
Biomni: A general-purpose biomedical ai agent
Huang, K., Zhang, S., Wang, H., Qu, Y., Lu, Y., Roohani, Y., Li, R., Qiu, L., Li, G., Zhang, J., et al. (2025) · doi:10.1101/2025.05.30.656746 · Cited by 125 · see citing works
Shi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., & Wang, M. D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1245 · Cited by 54 · see citing works
“…and efficiently mitigating error propagation and learning from environmental feedback (Qiao et al., 2023; Shinn et al., 2023; Yang et al., 2023 In such cases, it is generally preferable to acknowledge failure rather than generate an incorrect answer, as this could lead to an inaccurate diagnosis.”
BackgroundThe AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
Yamada, Y., Lange, R. T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J., & Ha, D. (2025) · arXiv · doi:10.48550/arxiv.2504.08066 · Cited by 32 · see citing works
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31 · see citing works
Unveiling Privacy Risks in LLM Agent Memory
Wang, B., He, W., Zeng, S., Xiang, Z., Xing, Y., Tang, J., & He, P. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1227 · Cited by 24 · see citing works
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24 · see citing works
Self-citationAgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents
Wang, H., Poskitt, C. M., & Sun, J. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3764546 · Cited by 14 · see citing works
Fang, J., Peng, Y., Zhang, X., Wang, Y., Yi, X., Zhang, G., Xu, Y., Wu, B., Liu, S., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.07407 · Cited by 6 · see citing works
Self-citationSWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.16941 · Cited by 4 · see citing works
“Wang et al. [17] introduced CodeAct, which unified agent action spaces using executable Python code, showing performance improvements of up to 20% over JSON or text-based approaches.”
Bai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., & Pei, J. (2026) · arXiv · doi:10.48550/arxiv.2604.22750 · Cited by 3 · see citing works
Self-citationSWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement
Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2410.20285 · Cited by 3 · see citing works
“Baselines Software agents leverage diverse tools, architectures, and models, leading to variability in their performance on subsets of the SWE-bench Lite dataset (Zhang et al., 2024a).”
AgentSquare: Automatic LLM Agent Search in Modular Design Space
Shang, Y., Li, Y., Zhao, K., Ma, L., Liu, J., Xu, F., & Li, Y. (2024) · arXiv · doi:10.48550/arxiv.2410.06153 · Cited by 3 · see citing works
Self-citationProgent: Securing AI Agents with Privilege Control
Shi, T., He, J., Wang, Z., Li, H., Wu, L., Guo, W., & Song, D. (2025) · arXiv · doi:10.48550/arxiv.2504.11703 · Cited by 3 · see citing works
Nedisan Videsjorden, A., Song, H., Goknil, A., Roman, D., & Soylu, A. (2026) · ACM Transactions on Internet of Things · doi:10.1145/3772077 · Cited by 5 · see citing works
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V., Li, E., Gupta, S., Sabharwal, A., & Balasubramanian, N. (2024) · arXiv · doi:10.48550/arxiv.2407.18901 · Cited by 2 · see citing works
FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
Dai, D., Liu, M., Li, A., Cao, J., Wang, Y., Wang, C., Peng, X., & Zheng, Z. (2025) · arXiv · doi:10.48550/arxiv.2504.06939 · Cited by 1 · see citing works
“While these systems demonstrate strong end-to-end performance, their success fundamentally relies on the model’s ability to internalize environmental and self-generated feedback.”
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., & Bui, N. D. Q. (2025) · arXiv · doi:10.48550/arxiv.2512.18470
“CodeAct (Wang et al., 2024c) unified agent action spaces through executable Python code, leading to performance gains compared to text-or JSON-based action formats.”
Lindenbauer, T., Slinko, I., Felder, L., Bogomolov, E., & Zharov, Y. (2025) · arXiv · doi:10.48550/arxiv.2508.21433
“The agent scaffolds we investigate use either ReAct [39] or CodeAct [29] as an agent framework.”
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
Yang, H., Yu, Q., & Desell, T. (2026) · arXiv · doi:10.48550/arxiv.2607.10569
“…of model reasoning for arithmetic and symbolic computation [8, 13]; CodeAct extended the move to the action layer of agentic systems, contrasting executable Python with JSON tool calls on general-purpose agent benchmarks [34], and the OpenHands platform [35] ships the CodeAct interface at scale.”
Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Feng, S., Wang, Z., Goyal, P., Wang, Y., Shi, W., Xia, H., Palangi, H., Zettlemoyer, L., Tsvetkov, Y., Lee, C.-Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.04510
“There are further risks in agentic [34, 104, 96, 72, 18, 31, 70, 115, 4, 10, 12, 14, 38, 42, 63, 81, 93, 101, 107, 114, 84, 2] versions of H ETEROGENEOUS S WARMS where one model could be biased or malicious.”
An Empirical Study of Harness Design for Coding Agents
Fan, R.-Z., Zhang, Z., Ma, S., Hu, Y., Wang, S., Song, K., Liu, F., Zamani, H., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2609.20804
“These choices are not incidental implementation details: changing the harness while holding the model fixed can substantially change model performance (Yang et al., 2024; Wang et al., 2024; Lewis, 2026).”
Self-citationReaComp: Compiling LLM Reasoning into Symbolic Solvers for Efficient Program Synthesis
Naik, A., Mathur, Y., Prakam, Rose, C., & Mortensen, D. (2026) · arXiv · doi:10.48550/arxiv.2605.05485
“CodeAct [Wang et al., 2024a] treats executable Python code as a unified action space for LLM agents, allowing actions to be executed and revised through an interpreter over multiple turns.”
Token Reduction Is Not Cost Reduction
Weinberger, S., & Hozez, A. (2026) · arXiv · doi:10.48550/arxiv.2607.12161 · Cited by 1 · see citing works
Finding the Right Fit: Model-Harness Interactions across Agent Tasks
Li, Y., Zhou, Y., Teng, Y. L., Yang, F., Deng, Y., Lyu, Z., Dong, X., Chen, F., & An, B. (2026) · arXiv
Self-citationSkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills
Zheng, B., Fatemi, M. Y., Jin, X., Wang, Z. Z., Gandhi, A., Song, Y., Gu, Y., Srinivasa, J., Liu, G., Neubig, G., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.07079
Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
Lumer, E., Nizar, F., Jangiti, A., Frank, K., Gulati, A., Phadate, M., & Subbiah, V. K. (2026) · arXiv · doi:10.48550/arxiv.2601.06007
Barbaste, P., Darrigol, T., Vu, G., & Wiltberger, T. (2026) · arXiv · doi:10.48550/arxiv.2609.00006
Evo-Bench: Can Language Models Improve Agent Harness?
Huang, L., Yang, C., Zhou, H., Song, H., Chen, Z., Le, R., Song, Y., Zhao, W. X., & Zhang, T. (2026) · arXiv · doi:10.48550/arxiv.2608.09096
Ning, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents
Xue, S., Zhong, J., Nan, Z., Li, W., Yu, Z., Ding, J., Gao, Q., Zhan, P., Zhang, Y., Cheng, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2609.17523
Self-citationSUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Bogin, B., Yang, K., Gupta, S., Richardson, K., Bransom, E., Clark, P., Sabharwal, A., & Khot, T. (2024) · arXiv · doi:10.48550/arxiv.2409.07440
- Authors
- Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., & Ji, H.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Apple (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2402.01030
- OpenAlex
- W4391555701
- Cited by
- 23
- References
- 0
- Citations by year
- 2026: 5 · 2025: 15 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2137603, 2138296, 2138307, 2138286)
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750, FA8650-23-C-7316)
- Office of Advanced Cyberinfrastructure (#2138286, #2138296, 2138259, #2138307, 2137603)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- LLM agents, Python interpreter, action space, multi-turn interaction, instruction-tuning dataset, tool invocation, Llama 2, Mistral
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
