Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.

JARVIS -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models

Wang, Zihao, Cai, Shaofei, Liu, Anji, Jin, Yonggang, Hou, Jinbing, Zhang, Bowei, Lin, Haowei, He, Zhaofeng, Zheng, Zilong, Yang, Yaodong, Ma, Xiaojian, Liang, Yitao · 2025 · IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract

Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduceJARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we developJARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfitJARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences.JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., “chopping trees” to long-horizon ones, e.g., “obtaining a diamond pickaxe”.JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task ofObtainDiamondPickaxe,JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. Furthermore, we show thatJARVIS-1 is able toself-improvefollowing a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

Overview

Integrity signals

Registry

Crossref · 10.1109/tpami.2024.3511593

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 2

  1. Fan, Building open-ended embodied agents with internet-scale knowledge
  2. Kanervisto, Minerl diamond 2021 competition: Overview, results, and lessons learned

Cited by

See all 41

Metadata

Authors
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., Ma, X., & Liang, Y.
Affiliations
  • Peking University (CN)
  • University of California, Los Angeles (US)
  • Beijing University of Posts and Telecommunications (CN)
  • Beijing Institute for General Artificial Intelligence (CN)
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2025-03
Volume
47
Issue
3
Pages
1894-1907
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
OpenAlex
W4405056274
Cited by
36
References
51
Citations by year
2026: 18 · 2025: 17
FWCI
5.33
Type
journal article
Language
en
Open access
closed
Locations
IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
Funders
  • National Key R&D Program of China (2022ZD0160301)
  • National Natural Science Foundation of China (62376031)
  • CCF-Tencent Rhino-Bird Open Research Fund
  • NSF (IIS-1943641, IIS-1956441, CCF-1837129)
  • SRA from Meta
  • research gift from Amazon Alexa AI
  • gift from Relational AI
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
multimodal language models, memory-augmented agents, Minecraft, embodied control, planning, long-horizon tasks, human development, lifelong learning, multimodal memory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.