JARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduceJARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we developJARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfitJARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences.JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., “chopping trees” to long-horizon ones, e.g., “obtaining a diamond pickaxe”.JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task ofObtainDiamondPickaxe,JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. Furthermore, we show thatJARVIS-1 is able toself-improvefollowing a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In multimodal settings, JARVIS1 (Wang et al., 2025q), EvoVLA (Liu et al., 2025i) and Auto-scaling Continuous Memory (Wu et al., 2025e) retain visual context, with the former storing survival experiences in Minecraft and the latter compressing GUI history into continuous embeddings.”
From Memory in the Age of AI Agents (2025) · cites this paper as Wang et al., 2025q
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“For better leveraging historical interaction between agent and environments, an explicit memory [41, 59] for more historical chatting has been leveraged for bigger storage of agent experiences.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Research works printed but not linked: 2
- Fan, Building open-ended embodied agents with internet-scale knowledge
- Kanervisto, Minerl diamond 2021 competition: Overview, results, and lessons learned
Cited by
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
ProAgent: Building Proactive Cooperative Agents with Large Language Models
Zhang, C., Yang, K., Hu, S., Wang, Z., Li, G., Sun, Y., Zhang, C., Zhang, Z., Liu, A., Zhu, S.-C., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29710 · Cited by 88
“Notable LLM-based agents like SayCan (Ahn et al. 2022), ReAct (Yao et al. 2023), DEPS (Wang et al. 2023b), RAP (Hao et al. 2023), Reflexion (Shinn et al. 2023), and JARVIS-1 (Wang et al. 2023c) have demonstrated the ability to make decisions interactively through appropriate prompts or feedback.”
BackgroundSelf-citationMMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.02483 · Cited by 65
Large multimodal agents: a survey
Xie, J., Chen, Z., Zhang, R., & Li, G. (2025) · Visual Intelligence · doi:10.1007/s44267-025-00093-y · Cited by 33
“Stored within the toolkit, each tool is designed for specific types of spatiotemporal reasoning and acts as an executor within the framework.”
InfluentialAppAgent: Multimodal Agents as Smartphone Users
Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z., Fu, B., & Yu, G. (2023) · arXiv · doi:10.48550/arxiv.2312.13771 · Cited by 11
“The emergence of multimodal LLM agents [3, 4, 11, 29, 36, 50], capable of processing various inputs including text, images, audio, and video, has further broadened the scope of LLM applications.”
KARMA: Augmenting Embodied AI Agents with Long-and-Short Term Memory Systems
Wang, Z., Yu, B., Zhao, J., Sun, W., Hou, S., Liang, S., Hu, X., Han, Y., & Gan, Y. (2025) · 2025 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra55743.2025.11128047 · Cited by 11
Self-citationFan, T., Wang, H., Zhao, Y., Zhou, S., & Shi, B. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104481 · Cited by 16
Scaling Instructable Agents Across Many Simulated Worlds
SIMA Team, Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield, B., Buttimore, G., Cant, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.10179 · Cited by 9
“…games that offer API access to the environment have served as a platform for grounding large language models (Wang et al., 2023a), and some works have even considered grounding a language model in a game through direct perception and action of a lower-level controller (Wang et al., 2023b).”
InfluentialSelf-citationLarge language models in human-robot collaboration: A systematic review, trends, and challenges
Yuan, G., Liu, X., Xiao, M., Xiao, J., & Wang, L. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.01.011 · Cited by 13
Lifelong Learning of Large Language Model Based Agents: A Roadmap
Zheng, J., Shi, C., Cai, X., Li, Q., Zhang, D., Li, C., Yu, D., & Ma, Q. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3650546 · Cited by 7
“For example, JARVIS-1 [49] improves its understanding of the environment through self-reflection and self-explanation, incorporating previous plans into its prompts.”
InfluentialAriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents
Anokhin, P., Semenov, N., Sorokin, A., Evseev, D., Kravchenko, A., Burtsev, M., & Burnaev, E. (2024) · arXiv · doi:10.48550/arxiv.2407.04363 · Cited by 8
“The Cooking game has the highest difficulty, because any error at intermediate step prevents completion of the whole game.”
Beyond Static GUI Agent: Evolving LLM-based GUI Testing via Dynamic Memory
Chen, M., Liu, Z., Chen, C., Wang, J., Xue, Y., Wu, B., Huang, Y., Wu, L., & Wang, Q. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00135 · Cited by 12
“Similarly, embodied agents in robotics and game environments [76]–[79] employ memory to capture past trajectories and outcomes, enabling agents to refine future decisions based on prior experiences.”
R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory
Huang, T., Basu, K., Abdelaziz, I., Kapanipathi, P., May, J., & Chen, M. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1464 · Cited by 4
Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory
Long, L., He, Y., Ye, W., Pan, Y., Lin, Y., Li, H., Zhao, J., & Li, W. (2025) · arXiv · doi:10.48550/arxiv.2508.09736 · Cited by 102
“A common approach is to append entire agent trajectories, such as dialogues [27, 31, 44, 59] or execution trajectories [14, 27, 29, 35, 36, 46], directly to memory.”
I-Lora: Iterative Merging of Routing-Tuned Low-Rank Adapters for Multi-Task Learning
Zhao, G., Zhang, Q., Zhai, S., Shen, D., Zhang, T., Qiao, Y., & Xu, T. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210099 · Cited by 3
“VLMs have been applied in gaming contexts [11], extended to act as agents planning complex actions [12], and to play games like Pok´emon by accessing external knowledge for in-context learning [13].”
范越, Ma, X., Su, R., Guo, J., Wu, R., Chen, X., & Li, Q. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00598 · Cited by 40
Self-citationPlaying NetHack with LLMs: Potential & Limitations as Zero-Shot Agents
Jeurissen, D., Perez-Liebana, D., Gow, J., Çakmak, D., & Kwan, J. (2024) · 2024 IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645630 · Cited by 2
“Recently, agents based on Large Language Models (LLMs) [1] have been successfully applied to robot environments [2], [3] and Minecraft [4]–[6], among others.”
BackgroundGenerative AI in Embodied Systems: System-Level Analysis of Performance, Efficiency and Scalability
Wan, Z.-S., Qian, J., Du, Y.-H., Jabbour, J., Du, Y.-L., Zhao, Y., Raychowdhury, A., Krishna, T., & Reddi, V. J. (2025) · IEEE International Symposium on Performance Analysis of Systems and Software · doi:10.1109/ispass64960.2025.00013 · Cited by 5
“Action memory [27]–[29] records the agents’ actions and status, and contains knowledge on how to execute specific high-level plans in different environments, encoded in the form of code or neural model parameters.”
InfluentialFan, Y., Ma, X., Su, R., Guo, J., Wu, R., Chen, X., & Li, Q. (2025) · arXiv · doi:10.48550/arxiv.2501.00358 · Cited by 2
Self-citationRodrigues, G., Rodrigues, K., Holanda, L. D. M. P., dos Santos, V., Benevides, L., Chagas, M., Alves, L. V., & Maia, P. H. M. (2025) · doi:10.5753/mssis.2025.16092 · Cited by 1
Self-balanced information bottleneck for few-shot multimodal information extraction
Wang, Y., Chen, K., Miao, Q., Liu, C., & Li, Y. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104985 · Cited by 1
Memory in the Age of AI Agents
Hu, Y., Liu, S., Yue, Y., Zhang, G., Liu, B., Zhu, F., Lin, J., Guo, H., Dou, S., Xi, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.13564 · Cited by 389
Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
Matthews, M., Beukman, M., Lu, C., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2410.23208 · Cited by 1
“Other notable work that aims to use open-ended discovery to train generalist agents include Voyager (Wang et al., 2024), Jarvis-1 (Wang et al., 2023) and Optimus-1 (Li et al., 2024b).”
Pecora, A. E., Strada, F., & Bottino, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97778-7_8 · Cited by 2
Multimodal hierarchical classification using cascade-of-thought
Hou, J., Tan, Z., Hu, Q., Wang, P., & Gong, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104555 · Cited by 1
Self-citationEvoVLA: Self-evolving Vision-Language-Action Model
Liu, Z., Yang, Z., Zhang, Z., & Tang, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37023-5_14 · Cited by 13
MMVPS: A Multimodal Large Model-Based Visual Perception System
Li, Z., Xie, X., Zeng, W., Yan, X., GuoFeng, L. U. O., Hou, X., Wang, C., Xu, L., & Chen, L. (2025) · IEEE Smart World Congress (SWC) · doi:10.1109/swc65939.2025.00180
Advance LLM-based Model-Context-Protocol and Agent to Agent Protocol for Child Medical Emergency
Farooq, M. S., Sarwar, M., Nawaz, N. A. H., Ahmed, U., Jamil, M. I., & Ahmad, J. (2025) · International Conference on Innovative Computing (ICIC) · doi:10.1109/icic68258.2025.11413209 · Cited by 1
BIFROST: An Adaptive Decision Framework for Regulating Depth of Thought in LLM Agents
Zhou, A. Y., Liu, B. Y., Xu, E. X., Li, C. Z., & Tian, D. J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463641
Tang, H., Zhang, F., Wan, K., Lu, S., Yao, Y., & Xu, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461447
Tang, H., Zhang, F., Wan, K., Lu, S., Zhang, X., Yao, Y., & Xu, X. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104830 · Cited by 2
Wang, Z., Dong, Y., Luo, F., Ruan, M., Cheng, Z., Chen, C., Li, P., & Liu, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00457 · Cited by 4
“Some other works focus on complex open worlds, such as Minecraft [19, 20, 29, 32, 37, 38], to evaluate model abilities of multimodal reasoning.”
Self-citationOpen-World Skill Discovery from Unsegmented Demonstration Videos
Deng, J.-W., Wang, Z., Cai, S., Liu, A., & Liang, Y.-T. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00997 · Cited by 5
Self-citationMa, R., Zhao, X., & Jin, L. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-4897-2_38
Chagas, M., Rodrigues, G., Rodrigues, K., Holanda, L. D. M. P., Alves, V. D. S., Benevides, L., Alves, L. V., & Maia, P. H. M. (2026) · IEEE/ACM International Workshop on Games and Software Engineering (GAS) · doi:10.1145/3786171.3788384
“For instance, the JARVIS-1 system explores Minecraft as an open-world environment for long-term simulations and continuous agent learning, leveraging multimodal models capable of perceiving, planning, and performing multiple tasks autonomously [13].”
Integrating Agglomerative Perception With One-Step Action Generation for Robotic Manipulation
Wang, S., Wang, L., Huo, H., Zhou, S., Xia, K., & Hua, G. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3713027
“Developing such agents remains challenging, as it requires reasoning over multimodal observations, including visual perception, 3D spatial structure, and language instructions [6], [7].”
Vatsal, S., Dubey, H., & Bajaj, A. (2026) · IEEE Access · doi:10.1109/access.2026.3720396
“JARVIS-1 [71] equips the agent with a multimodal memory that stores game experiences and retrieves them to guide later planning.”
Designing a Modular Agentic AI as a Personal Assistant for Grid Computing System
Ghosh, A., Kundu, S., Saha, D., Biswas, A., Das, P., Jana, S., & Chakraborty, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-25446-7_2
A Survey of Multi-Model Collaboration in Video Understanding
Chen, Y., Zhang, J., Zhang, L., Liu, C., Gao, R., Lu, Z., Qi, J., & Lei, Q. (2026) · Data · doi:10.3390/data11090230
My-Meetings: LLM-Based Multi-agent Conference Scene Designer
Cheng, X., & Feng, G. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3991-7_16
Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
Du, P. (2026) · arXiv · Cited by 1
“JARVIS-1 [Wang et al., 2024b] extends the hierarchical principle to multimodal settings, with separate stores for visual observations, textual plans, and executable skills.”
InfluentialA Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents
Huang, W.-C., Zhang, W., Liang, Y., Bei, Y., Chen, Y., Feng, T., Pan, X., Tan, Z., Wang, Y., Wei, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.06052 · Cited by 70
Self-citation
Metadata
- Authors
- Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., Ma, X., & Liang, Y.
- Affiliations
- Peking University (CN)
- University of California, Los Angeles (US)
- Beijing University of Posts and Telecommunications (CN)
- Beijing Institute for General Artificial Intelligence (CN)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2025-03
- Volume
- 47
- Issue
- 3
- Pages
- 1894-1907
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4405056274
- Cited by
- 36
- References
- 51
- Citations by year
- 2026: 18 · 2025: 17
- FWCI
- 5.33
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- Funders
- National Key R&D Program of China (2022ZD0160301)
- National Natural Science Foundation of China (62376031)
- CCF-Tencent Rhino-Bird Open Research Fund
- NSF (IIS-1943641, IIS-1956441, CCF-1837129)
- SRA from Meta
- research gift from Amazon Alexa AI
- gift from Relational AI
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- multimodal language models, memory-augmented agents, Minecraft, embodied control, planning, long-horizon tasks, human development, lifelong learning, multimodal memory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.