Gibson Env: Real-World Perception for Embodied Agents
Open accessAbstract
Developing visual perception models for active agents and sensorimotor control in the physical world are cumbersome as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we investigate developing real-world perception for active agents, propose Gibson Environment for this purpose, and showcase a set of perceptual tasks learned therein. Gibson is based upon virtualizing real spaces, rather than artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism "Goggles" enabling deploying the trained models in real-world without needing domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many works on scene design learn spatial knowledge priors from existing 3D scene databases or leverage user input and refine the 3D scene iteratively.
“Many works on scene design either learn spatial knowledge priors from existing 3D scene databases [4, 31, 49-51, 54, 59] or leverage user input and refine the 3D scene iteratively [3, 5].”
From Holodeck: Language Guided Generation of 3D Embodied AI Environments (2024) · cites this paper as [51]
Our method demonstrates obvious improvement compared to the previous method that applies very strong pre-trained visual representations.
“Compared to the previous method, which applies very strong pre-trained visual representations [32, 81], our method still demonstrates obvious improvement”
From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [81]
The citation lists haven’t been fetched yet
References 113 · Cited by 763
- Authors
- Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 9068-9079
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2953127211
- Cited by
- 763
- References
- 113
- Citations by year
- 2026: 51 · 2025: 100 · 2024: 115 · 2023: 105 · 2022: 96 · 2021: 139 · 2020: 90 · 2019: 59 · 2018: 8
- FWCI
- 30.31
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Multidisciplinary University Research Initiative
- Office of Naval Research (N000141410671, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- embodied agents, learning from simulation, semantic scene understanding, domain adaptation, sensorimotor control, field deployment, virtual buildings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
