CoChat
Sign Up
PreprintCited by 16AI-based Problem Solving and PlanningTopic ModelingNatural Language Processing Techniques

TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage

Open access
Ruan, Jingqing, Chen, Yihong, Zhang, Bin, Xu, Zhiwei, Bao, Tianpeng, Du, Guoqing, Shi, Shiwei, Mao, Hangyu +3 more·2023·arXiv
DOI 10.48550/arxiv.2308.03427arXiv 2308.03427
Open this paper in CoChat — free accountOpen PDF

Abstract

With recent advancements in natural language processing, Large Language Models (LLMs) have emerged as powerful tools for various real-world applications. Despite their prowess, the intrinsic generative abilities of LLMs may prove insufficient for handling complex tasks which necessitate a combination of task planning and the usage of external tools. In this paper, we first propose a structured framework tailored for LLM-based AI Agents and discuss the crucial capabilities necessary for tackling intricate problems. Within this framework, we design two distinct types of agents (i.e., one-step agent and sequential agent) to execute the inference process. Subsequently, we instantiate the framework using various LLMs and evaluate their Task Planning and Tool Usage (TPTU) abilities on typical tasks. By highlighting key findings and challenges, our goal is to provide a helpful resource for researchers and practitioners to leverage the power of LLMs in their AI applications. Our study emphasizes the substantial potential of these models, while also identifying areas that need more investigation and improvement.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.03427

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The AI Agents framework based on LLMs possesses the critical capabilities required to solve complex problems.

    “The AI Agents framework based on LLMs possesses the critical capabilities required to solve complex problems (Ruan et al. 2023).”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Ruan et al. 2023

    Section 2.2 (p. 4) identifies task planning and tool usage as abilities LLM-based agents need for complex tasks, and the Conclusion (p. 16) says the framework helps evaluate those abilities. The paper does not establish that the framework already possesses all the critical capabilities needed to solve complex problems; it also identifies weaknesses that require improvement. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 92

  1. [1][1] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al. , 'A survey of large language models,' arXiv preprint arXiv:2303.18223 , 2023. - · arXiv
  2. [2][2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , 'Language models are few-shot learners,' Advances in neural information processing systems , vol. 33, pp. 1877-1901, 2020. -
  3. [3][3] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, 'Finetuned language models are zero-shot learners,' arXiv preprint arXiv:2109.01652 , 2021. - · arXiv
  4. [5][5] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, 'Toolformer: Language models can teach themselves to use tools,' arXiv preprint arXiv:2302.04761 , 2023. - · arXiv
  5. [6][6] N. R. Jennings, K. Sycara, and M. Wooldridge, 'A roadmap of agent research and development,' Autonomous agents and multi-agent systems , vol. 1, pp. 7-38, 1998. -
  6. [7][7] N. R. Jennings and M. Wooldridge, 'Applying agent technology,' Applied Artificial Intelligence an International Journal , vol. 9, no. 4, pp. 357-369, 1995. -
  7. [8][8] S. Franklin and A. Graesser, 'Is it an agent, or just a program?: A taxonomy for autonomous agents,' in International workshop on agent theories, architectures, and languages . Springer, 1996, pp. 21-35. -
  8. [9][9] C. Castelfranchi, 'Modelling social action for ai agents,' Artificial intelligence , vol. 103, no. 1-2, pp. 157-182, 1998. -
  9. [10][10] J. Ferber and G. Weiss, Multi-agent systems: an introduction to distributed artificial intelligence . Addison-wesley Reading, 1999, vol. 1. -
  10. [11][11] L. Panait and S. Luke, 'Cooperative multi-agent learning: The state of the art,' Autonomous agents and multi-agent systems , vol. 11, pp. 387-434, 2005. -
  11. [12][12] M. Pourreza and D. Rafiei, 'Din-sql: Decomposed in-context learning of text-to-sql with self-correction,' arXiv preprint arXiv:2304.11015 , 2023. - · arXiv
  12. [13][13] C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, 'Visual chatgpt: Talking, drawing and editing with visual foundation models,' arXiv preprint arXiv:2303.04671 , 2023. - · arXiv
  13. [14][14] J. Gorniak, Y. Kim, S. Gwon, D. Wei, and N. W. Kim, 'Vizability: Multimodal accessible data visualization with keyboard navigation and conversational interaction,' arXiv preprint arXiv:2310.09611 , 2023. - · arXiv
  14. [16][16] Q. Xu, F. Hong, B. Li, C. Hu, Z. Chen, and J. Zhang, 'On the tool manipulation capability of open-source large language models,' arXiv preprint arXiv:2305.16504 , 2023. - · arXiv
  15. [17][17] Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian et al. , 'Toolllm: Facilitating large language models to master 16000+ real-world apis,' arXiv preprint arXiv:2307.16789 , 2023. - · arXiv
  16. [18][18] M. Li, F. Song, B. Yu, H. Yu, Z. Li, F. Huang, and Y. Li, 'Api-bank: A benchmark for tool-augmented llms,' arXiv preprint arXiv:2304.08244 , 2023. - · arXiv
  17. [19][19] S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, 'Gorilla: Large language model connected with massive apis,' arXiv preprint arXiv:2305.15334 , 2023. - · arXiv
  18. [20][20] Q. Tang, Z. Deng, H. Lin, X. Han, Q. Liang, and L. Sun, 'Toolalpaca: Generalized tool learning for language models with 3000 simulated cases,' arXiv preprint arXiv:2306.05301 , 2023. - · arXiv
  19. [21][21] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , 'Training language models to follow instructions with human feedback,' Advances in Neural Information Processing Systems , vol. 35, pp. 27 730-27 744, 2022. -
  20. [22][22] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon et al. , 'Constitutional ai: Harmlessness from ai feedback,' arXiv preprint arXiv:2212.08073 , 2022. - · arXiv
  21. [23][23] A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia et al. , 'Glm-130b: An open bilingual pre-trained model,' arXiv preprint arXiv:2210.02414 , 2022. - · arXiv
  22. [24][24] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , 'Llama: Open and efficient foundation language models,' arXiv preprint arXiv:2302.13971 , 2023. - · arXiv
  23. [25][25] Y. Cui, Z. Yang, and X. Yao, 'Efficient and effective text encoding for chinese llama and alpaca,' arXiv preprint arXiv:2304.08177 , 2023. - · arXiv
  24. [26][26] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, 'Chain-of-thought prompting elicits reasoning in large language models,' Neural Information Processing Systems , 2022. -
  25. [27][27] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , 'On the opportunities and risks of foundation models,' arXiv preprint arXiv:2108.07258 , 2021. - · arXiv
  26. [28][28] M. Mosbach, T. Pimentel, S. Ravfogel, D. Klakow, and Y. Elazar, 'Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation,' arXiv preprint arXiv:2305.16938 , 2023. - · arXiv
  27. [29][29] J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, and X. Hu, 'Harnessing the power of llms in practice: A survey on chatgpt and beyond,' arXiv preprint arXiv:2304.13712 , 2023. - · arXiv
  28. [30][30] C. Zhang, C. Zhang, C. Li, Y. Qiao, S. Zheng, S. K. Dam, M. Zhang, J. U. Kim, S. T. Kim, J. Choi et al. , 'One small step for generative ai, one giant leap for agi: A complete survey on chatgpt in aigc era,' arXiv preprint arXiv:2304.06488 , 2023. - · arXiv
  29. [31][31] F. Yu, H. Zhang, and B. Wang, 'Nature language reasoning, a survey,' arXiv preprint arXiv:2303.14725 , 2023. - · arXiv
  30. [32][32] Z. Wang, G. Zhang, K. Yang, N. Shi, W. Zhou, S. Hao, G. Xiong, Y. Li, M. Y. Sim, X. Chen et al. , 'Interactive natural language processing,' arXiv preprint arXiv:2305.13246 , 2023. - · arXiv
  31. [33][33] Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, Y. Huang, C. Xiao, C. Han et al. , 'Tool learning with foundation models,' arXiv preprint arXiv:2304.08354 , 2023. - · arXiv
  32. [34][34] W. Yu, C. Zhu, Z. Li, Z. Hu, Q. Wang, H. Ji, and M. Jiang, 'A survey of knowledge-enhanced text generation,' ACM Computing Surveys , vol. 54, no. 11s, pp. 1-38, 2022. -
  33. [35][35] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, 'Lora: Low-rank adaptation of large language models,' arXiv preprint arXiv:2106.09685 , 2021. - · arXiv
  34. [36][36] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, 'Parameter-efficient transfer learning for nlp,' in International Conference on Machine Learning . PMLR, 2019, pp. 2790-2799. -
  35. [37][37] X. L. Li and P. Liang, 'Prefix-tuning: Optimizing continuous prompts for generation,' arXiv preprint arXiv:2101.00190 , 2021. - · arXiv
  36. [38][38] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang, 'Gpt understands, too,' arXiv preprint arXiv:2103.10385 , 2021. - · arXiv
  37. [39][39] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, 'React: Synergizing reasoning and acting in language models,' arXiv preprint arXiv:2210.03629 , 2022. - · arXiv
  38. [40][40] T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal, 'Decomposed prompting: A modular approach for solving complex tasks,' arXiv preprint arXiv:2210.02406 , 2022. - · arXiv
  39. [41][41] G. Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz et al. , 'Augmented language models: a survey,' arXiv preprint arXiv:2302.07842 , 2023. - · arXiv
  40. [42][42] J. Duan, S. Yu, H. L. Tan, H. Zhu, and C. Tan, 'A survey of embodied ai: From simulators to research tasks,' IEEE Transactions on Emerging Topics in Computational Intelligence , vol. 6, no. 2, pp. 230-244, 2022. -
  41. [43][43] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik et al. , 'Habitat: A platform for embodied ai research,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 9339-9347. -
  42. [44][44] S. Franklin, 'Autonomous agents as embodied ai,' Cybernetics & Systems , vol. 28, no. 6, pp. 499-520, 1997. -
  43. [45][45] W. Zhang, Y. Guo, L. Niu, P. Li, C. Zhang, Z. Wan, J. Yan, F. U. D. Farrukh, and D. Zhang, 'Lp-slam: Language-perceptive rgb-d slam system based on large language model,' arXiv preprint arXiv:2303.10089 , 2023. - · arXiv
  44. [46][46] D. Shah, B. Osi´ nski, S. Levine et al. , 'Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,' in Conference on Robot Learning . PMLR, 2023, pp. 492-504. -
  45. [47][47] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian et al. , 'Do as i can, not as i say: Grounding language in robotic affordances,' in Conference on Robot Learning . PMLR, 2023, pp. 287-318. -
  46. [48][48] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar et al. , 'Inner monologue: Embodied reasoning through planning with language models,' arXiv preprint arXiv:2207.05608 , 2022. - · arXiv
  47. [49][49] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler, 'Open-vocabulary queryable scene representations for real world planning,' in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 11 50911 522. -
  48. [50][50] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al. , 'Palm-e: An embodied multimodal language model,' arXiv preprint arXiv:2303.03378 , 2023. - · arXiv
  49. [51][51] N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi, 'Chatgpt empowered long-step robot control in various environments: A case application,' arXiv preprint arXiv:2304.03893 , 2023. - · arXiv
  50. [52][52] K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf, 'Sayplan: Grounding large language models using 3d scene graphs for scalable task planning,' arXiv preprint arXiv:2307.06135 , 2023. - · arXiv
  51. [53][53] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su, 'Llm-planner: Fewshot grounded planning for embodied agents with large language models,' arXiv preprint arXiv:2212.04088 , 2022. - · arXiv
  52. [54][54] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , 'Rt-1: Robotics transformer for real-world control at scale,' arXiv preprint arXiv:2212.06817 , 2022. - · arXiv
  53. [55][55] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn et al. , 'Open-world object manipulation using pre-trained vision-language models,' arXiv preprint arXiv:2303.00905 , 2023. - · arXiv
  54. [56][56] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg et al. , 'A generalist agent,' arXiv preprint arXiv:2205.06175 , 2022. - · arXiv
  55. [57][57] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, 'Chatgpt for robotics: Design principles and model abilities,' Microsoft Auton. Syst. Robot. Res , vol. 2, p. 20, 2023. -
  56. [58][58] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, 'Code as policies: Language model programs for embodied control,' in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 9493-9500. -
  57. [59][59] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, 'Retrieval augmented language model pre-training,' in International conference on machine learning . PMLR, 2020, pp. 3929-3938. -
  58. [60][60] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al. , 'Retrieval-augmented generation for knowledge-intensive nlp tasks,' Advances in Neural Information Processing Systems , vol. 33, pp. 9459-9474, 2020. -
  59. [61][61] S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark et al. , 'Improving language models by retrieving from trillions of tokens,' in International conference on machine learning . PMLR, 2022, pp. 2206-2240. -
  60. [62][62] A. Sridhar, R. Lo, F. F. Xu, H. Zhu, and S. Zhou, 'Hierarchical prompting assists large language model on web navigation,' arXiv preprint arXiv:2305.14257 , 2023. - · arXiv
  61. [63][63] H. Furuta, O. Nachum, K.-H. Lee, Y. Matsuo, S. S. Gu, and I. Gur, 'Multimodal web navigation with instruction-finetuned foundation models,' arXiv preprint arXiv:2305.11854 , 2023. - · arXiv
  62. [64][64] Y. Qin, Z. Cai, D. Jin, L. Yan, S. Liang, K. Zhu, Y. Lin, X. Han, N. Ding, H. Wang et al. , 'Webcpm: Interactive web search for chinese long-form question answering,' arXiv preprint arXiv:2305.06849 , 2023. - · arXiv
  63. [65][65] S. Yao, H. Chen, J. Yang, and K. Narasimhan, 'Webshop: Towards scalable real-world web interaction with grounded language agents,' Advances in Neural Information Processing Systems , vol. 35, pp. 20 744-20 757, 2022. -
  64. [66][66] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders et al. , 'Webgpt: Browser-assisted question-answering with human feedback,' arXiv preprint arXiv:2112.09332 , 2021. - · arXiv
  65. [67][67] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, 'Hotpotqa: A dataset for diverse, explainable multi-hop question answering,' arXiv preprint arXiv:1809.09600 , 2018. - · arXiv
  66. [68][68] B. Wang, G. Li, and Y. Li, 'Enabling conversational interaction with mobile ui using large language models,' in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , 2023, pp. 1-17. -
  67. [70][70] H. Li, J. Su, Y . Chen, Q. Li, and Z. Zhang, 'Sheetcopilot: Bringing software productivity to the next level through large language models,' arXiv preprint arXiv:2305.19308 , 2023. - · arXiv
  68. [71][71] L. Zha, J. Zhou, L. Li, R. Wang, Q. Huang, S. Yang, J. Yuan, C. Su, X. Li, A. Su et al. , 'Tablegpt: Towards unifying tables, nature language and commands into one gpt,' arXiv preprint arXiv:2307.08674 , 2023. - · arXiv
  69. [72][72] Z. Chen, K. Zhou, B. Zhang, Z. Gong, W. X. Zhao, and J.-R. Wen, 'Chatcot: Toolaugmented chain-of-thought reasoning on \\ chat-based large language models,' arXiv preprint arXiv:2305.14323 , 2023. - · arXiv
  70. [73][73] A. Parisi, Y. Zhao, and N. Fiedel, 'Talm: Tool augmented language models,' arXiv preprint arXiv:2205.12255 , 2022. - · arXiv
  71. [74][74] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al. , 'Training verifiers to solve math word problems,' arXiv preprint arXiv:2110.14168 , 2021. - · arXiv
  72. [75][75] Z. Yang, L. Li, J. Wang, K. Lin, E. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and L. Wang, 'Mm-react: Prompting chatgpt for multimodal reasoning and action,' arXiv preprint arXiv:2303.11381 , 2023. - · arXiv
  73. [76][76] Z. Liu, Y. He, W. Wang, W. Wang, Y. Wang, S. Chen, Q. Zhang, Y. Yang, Q. Li, J. Yu et al. , 'Internchat: Solving vision-centric tasks by interacting with chatbots beyond language,' arXiv preprint arXiv:2305.05662 , 2023. - · arXiv
  74. [77][77] Y. Ge, W. Hua, J. Ji, J. Tan, S. Xu, and Y . Zhang, 'Openagi: When llm meets domain experts,' arXiv preprint arXiv:2304.04370 , 2023. - · arXiv
  75. [78][78] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, 'Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface,' arXiv preprint arXiv:2303.17580 , 2023. - · arXiv
  76. [79][79] D. Surís, S. Menon, and C. Vondrick, 'Vipergpt: Visual inference via python execution for reasoning,' arXiv preprint arXiv:2303.08128 , 2023. - · arXiv
  77. [80][80] T. Gupta and A. Kembhavi, 'Visual programming: Compositional visual reasoning without training,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14 953-14 962. -
  78. [81][81] L. Chen, B. Li, S. Shen, J. Yang, C. Li, K. Keutzer, T. Darrell, and Z. Liu, 'Language models are visual reasoning coordinators,' in ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2023. -
  79. [82][82] P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao, 'Chameleon: Plug-and-play compositional reasoning with large language models,' arXiv preprint arXiv:2304.09842 , 2023. - · arXiv
  80. [83][83] Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, and W. Chen, 'Critic: Large language models can self-correct with tool-interactive critiquing,' arXiv preprint arXiv:2305.11738 , 2023. - · arXiv
  81. [84][84] Y. Liang, C. Wu, T. Song, W. Wu, Y. Xia, Y. Liu, Y. Ou, S. Lu, L. Ji, S. Mao et al. , 'Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis,' arXiv preprint arXiv:2303.16434 , 2023. - · arXiv
  82. [85][85] S. Hao, T. Liu, Z. Wang, and Z. Hu, 'Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings,' arXiv preprint arXiv:2305.11554 , 2023. - · arXiv
  83. [86][86] B. Paranjape, S. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, and M. T. Ribeiro, 'Art: Automatic multi-step reasoning and tool-use for large language models,' arXiv preprint arXiv:2303.09014 , 2023. - · arXiv
  84. [87][87] G. Kim, P. Baldi, and S. McAleer, 'Language models can solve computer tasks,' arXiv preprint arXiv:2303.17491 , 2023. - · arXiv
  85. [88][88] T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou, 'Large language models as tool makers,' arXiv preprint arXiv:2305.17126 , 2023. - · arXiv
  86. [89][89] R. H. Lewis and J. Jiao, 'Computegpt: A computational chat model for numerical problems,' arXiv preprint arXiv:2305.06223 , 2023. - · arXiv
  87. [90][90] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, 'Pal: Programaided language models,' in International Conference on Machine Learning . PMLR, 2023, pp. 10 764-10 799. -
  88. [91][91] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, 'Voyager: An open-ended embodied agent with large language models,' arXiv preprint arXiv:2305.16291 , 2023. - · arXiv
  89. [92][92] C. Qian, C. Han, Y. R. Fung, Y. Qin, Z. Liu, and H. Ji, 'Creator: Disentangling abstract and concrete reasonings of large language models through tool creation,' arXiv preprint arXiv:2305.14318 , 2023. - · arXiv
  90. [93][93] Y. Cai, S. Mao, W. Wu, Z. Wang, Y. Liang, T. Ge, C. Wu, W. You, T. Song, Y. Xia et al. , 'Low-code llm: Visual programming over llms,' arXiv preprint arXiv:2304.08103 , 2023. - · arXiv
  91. [94][94] S. Arora, B. Yang, S. Eyuboglu, A. Narayan, A. Hojel, I. Trummer, and C. Ré, 'Language models enable simple systems for generating structured views of heterogeneous data lakes,' arXiv preprint arXiv:2304.09433 , 2023. - · arXiv
  92. [95][95] W. Zhang, Y. Shen, W. Lu, and Y. Zhuang, 'Data-copilot: Bridging billions of data and humans with autonomous workflow,' arXiv preprint arXiv:2306.07209 , 2023. · arXiv

Cited by

Sorted by citations

Authors
Ruan, J., Chen, Y., Zhang, B., Xu, Z., Bao, T., Du, G., Shi, S., Mao, H., Li, Z., Zeng, X., & Zhao, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2308.03427
arXiv
2308.03427
OpenAlex
W4385681800
Cited by
11
References
1
Citations by year
2026: 4 · 2025: 4 · 2024: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, AI agents, task planning, tool use, agent frameworks, external tools, multi-step reasoning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review