Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions
Open accessAbstract
Auto-GPT is an autonomous agent that leverages recent advancements in adapting Large Language Models (LLMs) for decision-making tasks. While there has been a growing interest in Auto-GPT stypled agents, questions remain regarding the effectiveness and flexibility of Auto-GPT in solving real-world decision-making tasks. Its limited capability for real-world engagement and the absence of benchmarks contribute to these uncertainties. In this paper, we present a comprehensive benchmark study of Auto-GPT styled agents in decision-making tasks that simulate real-world scenarios. Our aim is to gain deeper insights into this problem and understand the adaptability of GPT-based agents. We compare the performance of popular LLMs such as GPT-4, GPT-3.5, Claude, and Vicuna in Auto-GPT styled decision-making tasks. Furthermore, we introduce the Additional Opinions algorithm, an easy and effective method that incorporates supervised/imitation-based learners into the Auto-GPT scheme. This approach enables lightweight supervised learning without requiring fine-tuning of the foundational LLMs. We demonstrate through careful baseline comparisons and ablation studies that the Additional Opinions algorithm significantly enhances performance in online decision-making benchmarks, including WebShop and ALFWorld.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM-powered agents often struggle to keep a grounded sense of reality in their actions and reasoning.
“LLM-powered agents often struggle to maintain a grounded sense of reality in their actions and reasoning (Figure 2).”
From Project Sid: Many-agent simulations toward AI civilization (2024) · cites this paper as [57]
The Introduction (page unavailable) identifies limited control over LLM agents’ behavior. Section 3.4.2 (page unavailable) reports that GPT-3.5 was misled by irrelevant advice in ALFWorld, affecting its actions, but contrasts this with GPT-4’s ability to use environmental data and reject such advice. These findings support a narrower difficulty in some agents’ decision-making, not the broad claim that LLM-powered agents often lose a grounded sense of reality in their actions and reasoning. (LLM, conf=0.78)
Experimental — accuracy not yet measuredAutoGPT and BabyAGI use GPT-4 as both a planner and executor: the model analyzes high-level objectives, decomposes them into actionable subtasks, invokes external APIs as needed, and monitors progress to determine subsequent actions.
“For instance, AutoGPT [37] and BabyAGI https://github.com/yoheinakajima/babyagi use GPT-4 as both a planner and executor: the model analyzes high-level objectives, decomposes them into actionable subtasks, invokes external APIs as needed, and monitors progress to determine subsequent actions.”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [37]
The Introduction describes Auto-GPT-style agents as receiving high-level goals, generating a plan for each action, and integrating tools; an ALFWorld example shows a GPT-4 agent choosing a command after assessing the task and its current observation. The paper tests GPT-4 alongside other models, however, so it does not establish that AutoGPT necessarily uses GPT-4. The BabyAGI claim has its own link rather than this citation. (LLM, conf=0.84)
Experimental — accuracy not yet measuredTool-augmented LLM-powered AI Agents have demonstrated potentials across a range of applications.
“Toolaugmented LLM-powered AI Agents have demonstrated potentials across a range of applications.”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [37]
Reasoning in agents like AutoGPT is enhanced with function-calling and prompt chaining to simulate thought processes.
“Reasoning in agents like AutoGPT [37] is enhanced with function-calling”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 23
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, and Cameron McKinnon. 2022. Constitutional AI: Harmlessness from AI Feedback. In arXiv:2212.08073. arXiv. https://ar5iv.org/abs/2212.08073 · arXiv
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022). · arXiv
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). · arXiv
- Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-Based Prompting for Multi-Step Reasoning. arXiv:2210.00720 [cs.CL] · arXiv
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019). · arXiv
- Yang Li, Zhi-Ping Cai, and Hong Xu. 2018. LLMP: exploiting LLDP for latency measurement in software-defined data center networks. Journal of Computer Science and Technology 33 (2018), 277-285.
- Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. 2023. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434 (2023). · arXiv
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019). · arXiv
- OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] · arXiv
- Joon Sung Park, Joseph C. O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. 2023. Generative Agents: Interac- tive Simulacra of Human Behavior. https://arxiv.org/pdf/2304.03442. pdf · arXiv
- Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large Language Model Connected with Massive APIs. arXiv:2305.15334 [cs.CL] · arXiv
- Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. 2011. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. arXiv preprint arXiv:1011.0686 (2011). arXiv:1011.0686 · arXiv
- Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace. arXiv:2303.17580 [cs.CL] · arXiv
- Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366 (2023). · arXiv
- Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Win- son Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. arXiv preprint arXiv:1912.01734 (2020). · arXiv
- Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. AlfWorld: Align- ing Text and Embodied Environments for Interactive Learning. arXiv preprint arXiv:2010.03768 (2021). · arXiv
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sha- ran Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self- Consistency Improves Chain of Thought Reasoning in Language Mod- els. arXiv:2203.11171 [cs.CL] · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022). · arXiv
- Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. 2023. Foundation Models for Decision Making: Problems, Methods, and Opportunities. arXiv e-prints (2023), arXiv-2303.
- Ilan Yaniv. 2004. The benefit of additional opinions. Current directions in psychological science 13, 2 (2004), 75-78.
- Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. arXiv preprint arXiv:2207.01206 (2022). · arXiv
- Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022). · arXiv
- VOLO Hero Microfiber Hair Towel, Super Absorbent, Ultra-Soft, Fast Drying | Reduce Dry Time by 50%, Large, Premium Wrap Towel, Sustainable Packaging, Anti-Frizz, Anti-Breakage, Hands-Free (Luna Gray) $34.99 loop number:2 Here's one suggestion for the command: click with 'back to search'. Please use this suggestion as a reference and make your own judgement. Determine which next command to use, and respond using the JSON format specified above: { "thoughts": { "text": "I found the matching item with a price lower than $20.", "reasoning": "The first item (B08G14B779) matches the description and has a price of $4.99, which is lower than $20.", "plan": "-click on the matching item (B08G14B779)\n-review the item details\n-make the purchase", "criticism": "Clicking 'back to search' is not necessary since I already found the matching item." }, "command": { "name": "click", "args": { "tool_input": "B08G14B779" } } }
Cited by
Sorted by citations
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
“Vid2Seq augments language models with specialized temporal markers, facilitating the prediction of event boundaries and textual descriptions within a unified output sequence [ Yang et al. , 2023a ] .”
BackgroundMethodInfluentialAI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 244 · see citing works
“In the wake of this breakthrough, the AI landscape underwent a rapid transformation, shifting from the use of standalone LLMs toward more autonomous, task-oriented frameworks [29].”
InfluentialAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186 · see citing works
Evaluating large language models as agents in the clinic
Mehandru, N., Miao, B. Y., Almaraz, E. R., Sushil, M., Butte, A. J., & Alaa, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01083-y · Cited by 143 · see citing works
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90 · see citing works
A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist
Zhang, W., Zhao, L., Xia, H., Sun, S., Sun, J., Qin, M., Li, X., Zhao, Y., Zhao, Y., Cai, X., et al. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671801 · Cited by 74 · see citing works
Loya, M., Sinha, D., & Futrell, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.241 · Cited by 33 · see citing works
Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions
Gridach, M., Nanavati, J., Abidine, K. Z. E., Mendes, L., & Mack, C. (2025) · arXiv · doi:10.48550/arxiv.2503.08979 · Cited by 18 · see citing works
Project Sid: Many-agent simulations toward AI civilization
AL, A., Ahn, A., Becker, N., Carroll, S., Christie, N., Cortes, M., Demirci, A., Du, M., Li, F., Luo, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.00114 · Cited by 12 · see citing works
AppAgent: Multimodal Agents as Smartphone Users
Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z., Fu, B., & Yu, G. (2023) · arXiv · doi:10.48550/arxiv.2312.13771 · Cited by 11 · see citing works
“Initiatives like AutoGPT [45], HuggingGPT [30], and MetaGPT [15] illustrate this trend, and many projects demonstrate impressive capabilities, moving beyond basic language tasks to engaging in activities requiring higher cognitive functions, such as software development [6, 27] and gaming [10, 26,…”
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents
Su, H., Luo, J., Liu, C., Yang, X., Zhang, Y., Dong, Y., & Zhu, J. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3688650 · Cited by 11 · see citing works
Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools
Wu, J., Zhu, J., Liu, Y., Xu, M., & Jin, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.04644 · Cited by 31 · see citing works
TradingAgents: Multi-Agents LLM Financial Trading Framework
Xiao, Y., Sun, E., Luo, D., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2412.20138 · Cited by 6 · see citing works
“Most existing LLM-based agent frameworks use natural language as the primary communication interface, typically through structured message histories or collections of agent-generated messages (Fatouros et al., 2024b; Li et al., 2023a; Yang et al., 2023c, 2024).”
Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., & Zhou, B. (2024) · arXiv · doi:10.48550/arxiv.2407.08940 · Cited by 5 · see citing works
“…agents are equipped with reasoning (Wei et al., 2023a; Yao et al., 2023a), planning (Shen et al., 2023; Valmeekam et al., 2023), decision-making (Yang et al., 2023a; Kang et al., 2023), and long-term memory capabilities (Zhu et al., 2023a; Hu et al., 2023), and they are constructed upon the…”
BackgroundDesign Patterns for Securing LLM Agents against Prompt Injections
Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., Fischer, M., Froelicher, D., Grosse, K., Naeff, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.08837 · Cited by 2 · see citing works
“Such agents have access to powerful tools (e.g., code execution (Yang et al., 2023)) and are expected to solve arbitrarily complex tasks while interacting with untrusted third-party data.”
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wolczyk, M., Khan, A., Pignatelli, E., Kuciński, Ł., Pinto, L., Fergus, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.13543 · Cited by 2 · see citing works
Escher-Loop: Mutual Evolution by Closed-Loop Self-Referential Optimization
Liu, Z., Guo, X., Wei, X., Hao, H., & Yang, L. (2026) · arXiv · doi:10.48550/arxiv.2604.23472
- Authors
- Yang, H., Yue, S., & He, Y.
- Affiliations
- Amazon (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.02224
- OpenAlex
- W4379539325
- Cited by
- 44
- References
- 0
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 19 · 2023: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- autonomous agents, online decision making, large language models, benchmarking, imitation learning, supervised learning, GPT-4, GPT-3.5, Claude, vicuña, WebShop, ALFWorld
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
