CoChat
Sign Up
PreprintCited by 24Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningRobot Manipulation and Learning

SEAL: Self-supervised embodied active learning using exploration and 3D consistency

Open access
Chaplot, Devendra Singh, Dalal, Murtaza, Gupta, Saurabh, Malik, Jitendra, Salakhutdinov, Ruslan·2021·arXiv
DOI 10.48550/arxiv.2112.01001arXiv 2112.01001
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we explore how we can build upon the data and models of Internet images and use them to adapt to robot vision without requiring any extra labels. We present a framework called Self-supervised Embodied Active Learning (SEAL). It utilizes perception models trained on internet images to learn an active exploration policy. The observations gathered by this exploration policy are labelled using 3D consistency and used to improve the perception model. We build and utilize 3D semantic maps to learn both action and perception in a completely self-supervised manner. The semantic map is used to compute an intrinsic motivation reward for training the exploration policy and for labelling the agent observations using spatio-temporal 3D consistency and label propagation. We demonstrate that the SEAL framework can be used to close the action-perception loop: it improves object detection and instance segmentation performance of a pretrained perception model by just moving around in training environments and the improved perception model can be used to improve Object Goal Navigation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2112.01001

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Perception models pre-trained on offline image datasets have a large domain gap and struggle with perception obstacles from rapid viewpoint changes.

    “Common perception models pre-trained on offline image datasets exhibit a significant domain gap and struggle to adapt to perception obstacles due to rapid changes in viewing perspectives.”

    From Unbiased Embodied Visual Representation Learning with Causal Inference and Cross-Modality Alignment (2025) · cites this paper as [5]

    The Introduction (p. 1) contrasts sparse Internet images with the continuous, changing views an embodied agent encounters, and the Method (p. 3) starts with a model pretrained on static Internet data. The paper supports the need to adapt that model to embodied perception, but it does not specifically attribute perception difficulties to rapid viewpoint changes or establish that the domain gap is significant. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 10

  1. [6][6] Ruzena Bajcsy. Active perception. Proceedings of the IEEE , 76(8):966-1005, 1988. 3 -
  2. [7][7] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 , pages 177-186. Springer, 2010. 6 -
  3. [9][9] Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In In Neural Information Processing Systems , 2020. 4, 5, 6, 8 -
  4. [10][10] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. In ICLR , 2020. 5, 7, 9 -
  5. [16][16] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1183-1192. JMLR. org, 2017. 3 -
  6. [19][19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV , pages 2961-2969, 2017. 1, 2, 3, 4, 6 -
  7. [20][20] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning , pages 1989-1998. PMLR, 2018. 3 -
  8. [21][21] Berthold Horn. Robot vision . MIT press, 1986. 1 -
  9. [43][43] James A Sethian. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences , 93(4):1591-1595, 1996. 5 -
  10. [54][54] Jianwei Yang, Zhile Ren, Mingze Xu, Xinlei Chen, David J Crandall, Devi Parikh, and Dhruv Batra. Embodied amodal recognition: Learning to move to perceive objects. In Proceedings of the IEEE International Conference on Computer Vision , pages 2040-2050, 2019. 3 -

Cited by

Sorted by citations

Related

Authors
Chaplot, D. S., Dalal, M., Gupta, S., Malik, J., & Salakhutdinov, R.
Affiliations
  • Meta (Israel) (IL)
  • Carnegie Mellon University (US)
  • Indraprastha Institute of Information Technology Delhi (IN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
34
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2112.01001
arXiv
2112.01001
OpenAlex
W3211613698
Cited by
22
References
0
Citations by year
2025: 3 · 2024: 5 · 2023: 12 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • neural information processing systems
Funders
  • National Aeronautics and Space Administration (80NSSC21K1030)
  • Nvidia
  • Carnegie Mellon University
  • University of Illinois at Urbana-Champaign
  • Stanford Bio-X
  • Multidisciplinary University Research Initiative
  • Defence Science and Technology Agency - Singapore
  • Office of Naval Research (N000141410671, N00014)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
self-supervised learning, robot vision, exploration policy, 3D consistency, intrinsic motivation, label propagation, perception-action loop, object detection, instance segmentation, object-goal navigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review