The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
JARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
Abstract
Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduceJARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we developJARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfitJARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences.JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., “chopping trees” to long-horizon ones, e.g., “obtaining a diamond pickaxe”.JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task ofObtainDiamondPickaxe,JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. Furthermore, we show thatJARVIS-1 is able toself-improvefollowing a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In multimodal settings, JARVIS1 (Wang et al., 2025q), EvoVLA (Liu et al., 2025i) and Auto-scaling Continuous Memory (Wu et al., 2025e) retain visual context, with the former storing survival experiences in Minecraft and the latter compressing GUI history into continuous embeddings.”
From Memory in the Age of AI Agents (2025) · cites this paper as Wang et al., 2025q
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“For better leveraging historical interaction between agent and environments, an explicit memory [41, 59] for more historical chatting has been leveraged for bigger storage of agent experiences.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
Self-citation
Research works printed but not linked: 2
- Fan, Building open-ended embodied agents with internet-scale knowledge
- Kanervisto, Minerl diamond 2021 competition: Overview, results, and lessons learned
Cited by
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
ProAgent: Building Proactive Cooperative Agents with Large Language Models
Zhang, C., Yang, K., Hu, S., Wang, Z., Li, G., Sun, Y., Zhang, C., Zhang, Z., Liu, A., Zhu, S.-C., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29710 · Cited by 88
“Notable LLM-based agents like SayCan (Ahn et al. 2022), ReAct (Yao et al. 2023), DEPS (Wang et al. 2023b), RAP (Hao et al. 2023), Reflexion (Shinn et al. 2023), and JARVIS-1 (Wang et al. 2023c) have demonstrated the ability to make decisions interactively through appropriate prompts or feedback.”
BackgroundSelf-citationLarge multimodal agents: a survey
Xie, J., Chen, Z., Zhang, R., & Li, G. (2025) · Visual Intelligence · doi:10.1007/s44267-025-00093-y · Cited by 259
“Stored within the toolkit, each tool is designed for specific types of spatiotemporal reasoning and acts as an executor within the framework.”
InfluentialAppAgent: Multimodal Agents as Smartphone Users
Zhang, C., Yang, Z., Liu, J., Li, Y., Han, Y., Chen, X., Huang, Z., Fu, B., & Yu, G. (2023) · arXiv · doi:10.48550/arxiv.2312.13771 · Cited by 11
“The emergence of multimodal LLM agents [3, 4, 11, 29, 36, 50], capable of processing various inputs including text, images, audio, and video, has further broadened the scope of LLM applications.”
KARMA: Augmenting Embodied AI Agents with Long-and-Short Term Memory Systems
Wang, Z., Yu, B., Zhao, J., Sun, W., Hou, S., Liang, S., Hu, X., Han, Y., & Gan, Y. (2025) · 2025 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra55743.2025.11128047 · Cited by 11
Self-citation
Metadata
- Authors
- Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., Ma, X., & Liang, Y.
- Affiliations
- Peking University (CN)
- University of California, Los Angeles (US)
- Beijing University of Posts and Telecommunications (CN)
- Beijing Institute for General Artificial Intelligence (CN)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2025-03
- Volume
- 47
- Issue
- 3
- Pages
- 1894-1907
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4405056274
- Cited by
- 36
- References
- 51
- Citations by year
- 2026: 18 · 2025: 17
- FWCI
- 5.33
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- Funders
- National Key R&D Program of China (2022ZD0160301)
- National Natural Science Foundation of China (62376031)
- CCF-Tencent Rhino-Bird Open Research Fund
- NSF (IIS-1943641, IIS-1956441, CCF-1837129)
- SRA from Meta
- research gift from Amazon Alexa AI
- gift from Relational AI
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- multimodal language models, memory-augmented agents, Minecraft, embodied control, planning, long-horizon tasks, human development, lifelong learning, multimodal memory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: JARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models. https://app.cochat.ai/papers/10.1109/tpami.2024.3511593
- The paper
- Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., Ma, X., & Liang, Y. (2025). JARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2024.3511593
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.