Navigating to objects in the real world
Open accessThe DOI matches its Crossref record; 106 of the 106 references it cites were found in a registry or an index.
Abstract
Semantic navigation is necessary to deploy mobile robots in uncontrolled environments such as homes or hospitals. Many learning-based approaches have been proposed in response to the lack of semantic understanding of the classical pipeline for spatial navigation, which builds a geometric map using depth sensors and plans to reach point goals. Broadly, end-to-end learning approaches reactively map sensor inputs to actions with deep neural networks, whereas modular learning approaches enrich the classical pipeline with learning-based semantic sensing and exploration. However, learned visual navigation policies have predominantly been evaluated in sim, with little known about what works on a robot. We present a large-scale empirical study of semantic visual navigation methods comparing representative methods with classical, modular, and end-to-end learning approaches across six homes with no prior experience, maps, or instrumentation. We found that modular learning works well in the real world, attaining a 90% success rate. In contrast, end-to-end learning does not, dropping from 77% sim to a 23% real-world success rate because of a large image domain gap between sim and reality. For practitioners, we show that modular learning is a reliable approach to navigate to objects: Modularity and abstraction in policy design enable sim-to-real transfer. For researchers, we identify two key issues that prevent today’s simulators from being reliable evaluation benchmarks—a large sim-to-real gap in images and a disconnect between sim and real-world error modes—and propose concrete steps forward.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Low-level navigation policies are typically derived from deterministic path planning conditioned on environmental constraints and high-level spatial goals52.”
From Brain-inspired spatial intelligence for embodied agents (2026) · cites this paper as [52]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 3×
“Depth features were computed with a ResNet50 pretrained on point goal navigation (44).”
MethodHuman-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundHe, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 4×
“RGB, depth, and semantic frames are then fed into a GRU [109] along with other low dimensional features — the robot pose, a learnable goal object embedding, the fraction of the visual input occupied by the goal category, and the previous action — to predict a distribution over next actions.”
BackgroundInfluentialContinuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundA robust layered control system for a mobile robot
Brooks, R. (1986) · IEEE Journal on Robotics and Automation · doi:10.1109/jra.1986.1087032 · Cited by 7,799
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 1×
“Among approaches that can operate in unseen environments, classical approaches typically build a geometric map of the environment using depth sensors [12, 13, 14], and later a monocular RGB camera [15, 16, 17], while simultaneously localizing the robot relative to its growing map.”
BackgroundMonoSLAM: Real-time single camera SLAM
Davison, A. J., Reid, I. D., Molton, N. D., & Stasse, O. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1049 · Cited by 3,959 · cited 1×
“Among approaches that can operate in unseen environments, classical approaches typically build a geometric map of the environment using depth sensors [12, 13, 14], and later a monocular RGB camera [15, 16, 17], while simultaneously localizing the robot relative to its growing map.”
BackgroundThe dynamic window approach to collision avoidance
Fox, D., Burgard, W., & Thrun, S. (1997) · IEEE Robotics & Automation Magazine · doi:10.1109/100.580977 · Cited by 3,771 · cited 1×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundA fast marching level set method for monotonically advancing fronts.
Sethian, J. A. (1996) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.93.4.1591 · Cited by 3,221 · cited 1×
“Planner Given a long-term goal output by the frontier or semantic exploration policy, we used the fast marching method (112) as in (63) to plan a path and the first low-level action along this path deterministically.”
MethodDomain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 1×
“Domain randomization [99] — randomizing environmental factors that are hard to simulate accurately during training to make policies robust to these factors — has been applied successfully in domains where randomization is straightforward like dexterous manipulation [100] where one needs to randomize over physics and s…”
BackgroundUsing occupancy grids for mobile robot perception and navigation
Elfes, A. (1989) · Computer · doi:10.1109/2.30720 · Cited by 2,213 · cited 1×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundSUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · cited 1×
“is a RedNet [110] pre-trained on the SUN RGB-D dataset [111] and fine-tuned on images from the Habitat simulator.”
MethodRobust Monte Carlo localization for mobile robots
Thrun, S., Fox, D., Burgard, W., & Dellaert, F. (2001) · Artificial Intelligence · doi:10.1016/s0004-3702(01)00069-8 · Cited by 1,809 · cited 3×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundMethodA frontier-based approach for autonomous exploration
Yamauchi, B. (2002) · Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA'97. 'Towards New Computational Principles for Robotics and Automation' · doi:10.1109/cira.1997.613851 · Cited by 1,695 · cited 3×
“Among the many heuristics for goal-agnostic exploration proposed in the classical navigation literature, we select frontier-based exploration [18] — navigate towards the closest unexplored region — to represent classical exploration methods because it was particularly effective in prior work [62].”
MethodInfluentialLearning dexterous in-hand manipulation
Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919887447 · Cited by 1,643 · cited 1×
“…factors that are hard to simulate accurately during training to make policies robust to these factors — has been applied successfully in domains where randomization is straightforward like dexterous manipulation [100] where one needs to randomize over physics and single object appearance.”
BackgroundTarget-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundSonar-based real-world mapping and navigation
Elfes, A. (1987) · IEEE Journal on Robotics and Automation · doi:10.1109/jra.1987.1087096 · Cited by 1,397
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340
Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Aparicio, J., & Goldberg, K. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.058 · Cited by 1,268 · cited 3×
“We should prioritize real-world transfer when designing policies: (A) replace policy architectures that directly operate on RGB images with ones leveraging abstractions as is common practice in other domains [56, 59, 89], and (B) avoid training a segmentation model on sim data if the policy architecture does not allow…”
BackgroundMethodHabitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 1×
“We can attribute this to (A) the emergence of sophisticated embodied simulators [78, 79, 35], which significantly reduced the field’s barrier to entry and sped up the proposal of new methods, and (B) the relative operational difficulties of bringing a robot out of the lab into diverse, realistic, deployment environmen…”
BackgroundSelf-citationAnderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
A flexible and scalable SLAM system with full 3D motion estimation
Kohlbrecher, S., von Stryk, O., Meyer, J., & Klingauf, U. (2011) · 2011 IEEE International Symposium on Safety, Security, and Rescue Robotics · doi:10.1109/ssrr.2011.6106777 · Cited by 1,086 · cited 1×
“We use off-the-shelf LiDAR-based Hector SLAM [104] to estimate the robot’s pose.”
MethodSupersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487517 · Cited by 1,047 · cited 1×
“It is the option of choice for perception stacks across robotics and has been applied to end-to-end control in domains where sub-optimal policy operation is safe, like grasping and static manipulation [97, 98].”
BackgroundSLAM++: Simultaneous Localisation and Mapping at the Level of Objects
Salas-Moreno, R. F., Newcombe, R. A., Strasdat, H., Kelly, P. H. J., & Davison, A. J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.178 · Cited by 945 · cited 1×
“Semantic SLAM methods [19, 20, 21, 22, 23, 24, 25] naturally extend SLAM to detect objects and keep them in memory but offer no solution for efficient semantic exploration.”
MethodExperiences with an interactive museum tour-guide robot
Burgard, W., Cremers, A. B., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., Steiner, W., & Thrun, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00070-3 · Cited by 818 · cited 2×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundLearning robust perceptive locomotion for quadrupedal robots in the wild
Miki, T., Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2022) · Science Robotics · doi:10.1126/scirobotics.abk2822 · Cited by 813 · cited 1×
“Our final option, training in simulation with modularity and abstraction (57)—designing abstractions of the input raw sensor data that contain sufficient information to solve the task while being invariant to environmental factors that are hard to simulate accurately—is the practice of choice in autonomous flight (91,…”
BackgroundGibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763
Self-citationA robot exploration and mapping strategy based on a semantic hierarchy of spatial representations
Kuipers, B., & Byun, Y.-T. (1991) · Robotics and Autonomous Systems · doi:10.1016/0921-8890(91)90014-c · Cited by 707 · cited 1×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundMINERVA: a second-generation museum tour-guide robot
Thrun, S., Bennewitz, M., Burgard, W., Cremers, A. B., Dellaert, F., Fox, D., Hahnel, D., Rosenberg, C., Roy, N., Schulte, J., et al. (2003) · Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C) · doi:10.1109/robot.1999.770401 · Cited by 701 · cited 2×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundBenchmarking 6DOF Outdoor Visual Localization in Changing Conditions
Sattler, T., Maddern, W., Toft, C., Torii, A., Hammarstrand, L., Stenborg, E., Safari, D., Okutomi, M., Pollefeys, M., Sivic, J., et al. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00897 · Cited by 699 · cited 1×
“Among approaches that can operate in unseen environments, classical approaches typically build a geometric map of the environment using depth sensors [12, 13, 14], and later a monocular RGB camera [15, 16, 17], while simultaneously localizing the robot relative to its growing map.”
BackgroundVector-based navigation using grid-like representations in artificial agents
Banino, A., Barry, C., Uria, B., Blundell, C., Lillicrap, T., Mirowski, P., Pritzel, A., Chadwick, M. J., Degris, T., Modayil, J., et al. (2018) · Nature · doi:10.1038/s41586-018-0102-6 · Cited by 677 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
Background(2020) · doi:10.1109/icra40945.2020.9196885 · Cited by 633 · cited 2×
“Semantic SLAM methods [19, 20, 21, 22, 23, 24] naturally extend SLAM to detect objects and keep them in memory in a spatial semantic map but offer no solution for efficient semantic exploration.”
MethodOn evaluation of embodied navigation agents
Anderson, P., Chang, A., Chaplot, D. S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R., Savva, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1807.06757 · Cited by 622 · cited 4×
“To compute the Success weighted by Path Length (SPL) [1], we measure the geodesic distance to the goal object instance closest to the starting location.”
MethodInfluentialSelf-citationBlanche-an experiment in guidance and navigation of an autonomous robot vehicle
Cox, I. J. (1991) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.75902 · Cited by 588 · cited 1×
“Many classical approaches to navigation require access to precomputed maps (7, 10, 11).”
Background6-DOF GraspNet: Variational Grasp Generation for Object Manipulation
Mousavian, A., Eppner, C., & Fox, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00299 · Cited by 571 · cited 2×
“Our final option, training in simulation with modularity and abstraction [56] — designing abstractions of the input raw sensor data that contain sufficient information to solve the task while being invariant to environmental factors that are hard to simulate accurately — is the practice of choice in autonomous flight…”
BackgroundPosition referencing and consistent world modeling for mobile robots
Chatila, R., & Laumond, J. (2005) · Proceedings. 1985 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.1985.1087373 · Cited by 567 · cited 1×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundDetecting Twenty-Thousand Classes Using Image-Level Supervision
Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., & Misra, I. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20077-9_21 · Cited by 564
Robust reconstruction of indoor scenes
Choi, S., Zhou, Q.-Y., & Koltun, V. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299195 · Cited by 555 · cited 19×
“A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, D. Batra, Habitat 2.0: Training home assistants to rearrange their habitat, in Advances in Neur…”
MethodInfluentialCognitive Mapping and Planning for Visual Navigation
Gupta, S., Davidson, J., Levine, S., Sukthankar, R., & Malik, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.769 · Cited by 500
Self-citationDD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion FramesPreprint
Wijmans, E., Kadian, A., Morcos, A. S., Lee, S., Essa, I. A., Parikh, D., Savva, M., & Batra, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00357 · Cited by 485 · cited 2×
“Depth features were computed with a ResNet50 pretrained on point goal navigation (44).”
MethodSelf-citationProbabilistic data association for semantic SLAM
Bowman, S. L., Atanasov, N., Daniilidis, K., & Pappas, G. J. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989203 · Cited by 462 · cited 2×
“Semantic SLAM methods [19, 20, 21, 22, 23, 24] naturally extend SLAM to detect objects and keep them in memory in a spatial semantic map but offer no solution for efficient semantic exploration.”
MethodPlaying FPS Games with Deep Reinforcement Learning
Lample, G., & Chaplot, D. S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10827 · Cited by 452 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundSelf-citationVisual-inertial navigation, mapping and localization: A scalable real-time causal approach
Jones, E. S., & Soatto, S. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364910388963 · Cited by 435 · cited 1×
“Among approaches that can operate in unseen environments, classical approaches typically build a geometric map of the environment using depth sensors [12, 13, 14], and later a monocular RGB camera [15, 16, 17], while simultaneously localizing the robot relative to its growing map.”
BackgroundA Survey of Embodied AI: From Simulators to Research Tasks
Duan, J., Yu, S., Tan, H. L., Zhu, H., & Tan, C. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3141105 · Cited by 420
Dynamic Map Building for an Autonomous Mobile Robot
Leonard, J. J., Durrant-Whyte, H. F., & Cox, I. J. (1992) · The International Journal of Robotics Research · doi:10.1177/027836499201100402 · Cited by 414 · cited 1×
“Navigation has been studied in robotics literature for over three decades [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].”
BackgroundLearning high-speed flight in the wild
Loquercio, A., Kaufmann, E., Ranftl, R., Müller, M., Koltun, V., & Scaramuzza, D. (2021) · Science Robotics · doi:10.1126/scirobotics.abg5810 · Cited by 375 · cited 2×
“We should prioritize real-world transfer when designing policies: (A) replace policy architectures that directly operate on RGB images with ones leveraging abstractions as is common practice in other domains [56, 59, 89], and (B) avoid training a segmentation model on sim data if the policy architecture does not allow…”
BackgroundMethodIQA: Visual Question Answering in Interactive Environments
Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., & Farhadi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00430 · Cited by 352
Learning to explore using active neural SLAM
Chaplot, D. S., Gandhi, D., Gupta, S., Gupta, A., & Salakhutdinov, R. (2020) · arXiv · doi:10.48550/arxiv.2004.05155 · Cited by 318 · cited 2×
“Modular learning can enable Sim-to-Real transfer [56, 61, 62]: one can shield learned modules from Sim-to-Real domain gaps by designing abstractions of the input raw sensor data that contain sufficient information to solve the task while being invariant to environmental factors that are hard to simulate accurately (li…”
BackgroundSelf-citationScaramuzza, D., Achtelik, M. C., Doitsidis, L., Friedrich, F., Kosmatopoulos, E., Martinelli, A., Achtelik, M. W., Chli, M., Chatzichristofis, S., Kneip, L., et al. (2014) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2014.2322295 · Cited by 299 · cited 1×
“cessful applications across robotics in autonomous driving [55, 56], flight [57], and grasping [58, 59, 60], much work has focused on designing modular learning approaches that aim to combine the benefits of both learning and classical methods.”
BackgroundMahler, J., Liang, J., Niyaz, S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09312 · Cited by 277 · cited 3×
“We should prioritize real-world transfer when designing policies: (A) replace policy architectures that directly operate on RGB images with ones leveraging abstractions as is common practice in other domains [56, 59, 89], and (B) avoid training a segmentation model on sim data if the policy architecture does not allow…”
BackgroundMethodLearning Dexterous In-Hand Manipulation
OpenAI, Andrychowicz, M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., et al. (2018) · arXiv · doi:10.48550/arxiv.1808.00177 · Cited by 248 · cited 1×
“…factors that are hard to simulate accurately during training to make policies robust to these factors — has been applied successfully in domains where randomization is straightforward like dexterous manipulation [100] where one needs to randomize over physics and single object appearance.”
BackgroundNeural Topological SLAM for Visual Navigation
Singh Chaplot, D., Salakhutdinov, R., Gupta, A., & Gupta, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01289 · Cited by 241 · cited 1×
“Representative examples of modular learning for exploration [61, 63] and for reaching object goals and image goals are [62, 64, 65, 66].”
BackgroundConcrete Problems for Autonomous Vehicle Safety: Advantages of Bayesian Deep Learning
McAllister, R., Gal, Y., Kendall, A., van der Wilk, M., Shah, A., Cipolla, R., & Weller, A. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/661 · Cited by 229
Kaufmann, E., Loquercio, A., Ranftl, R., Müller, M., Koltun, V., & Scaramuzza, D. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.040 · Cited by 222
Joint Semantic Segmentation and 3D Reconstruction from Monocular Video
Kundu, A., Li, Y., Dellaert, F., Li, F., & Rehg, J. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_45 · Cited by 220 · cited 2×
“Semantic SLAM methods [19, 20, 21, 22, 23, 24] naturally extend SLAM to detect objects and keep them in memory in a spatial semantic map but offer no solution for efficient semantic exploration.”
MethodOccupancy Anticipation for Efficient Exploration and Navigation
Ramakrishnan, S. K., Al-Halah, Z., & Grauman, K. L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_24 · Cited by 218
Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?
Kadian, A., Truong, J., Gokaslan, A., Clegg, A., Wijmans, E., Lee, S., Savva, M., Chernova, S., & Batra, D. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3013848 · Cited by 201 · cited 3×
“We compute the Sim-vs-Real Correlation Coefficient (SRCC) introduced in [93], which is the Pearson correlation between binary episode outcomes in sim vs.”
MethodInfluentialSelf-citationScene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Fang, K., Toshev, A., Fei-Fei, L., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00063 · Cited by 178
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A., & Wijmans, E. (2020) · arXiv · doi:10.48550/arxiv.2006.13171 · Cited by 177 · cited 1×
“We consider the problem of semantic navigation instantiated by the Object Goal task [62, 1, 91].”
MethodSelf-citationSimple but Effective: CLIP Embeddings for Embodied AI
Khandelwal, A., Weihs, L., Mottaghi, R., & Kembhavi, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01441 · Cited by 172
RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation
Jiang, J., Zheng, L., Luo, F., & Zhang, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.01054 · Cited by 172 · cited 2×
“is a RedNet [110] pre-trained on the SUN RGB-D dataset [111] and fine-tuned on images from the Habitat simulator.”
MethodEpisodic Curiosity through Reachability
Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T., & Gelly, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02274 · Cited by 170 · cited 1×
“…policies often directly map sensor data to actions, akin to a modern version of Rodney Brook’s Subsumption architecture [46], researchers have also introduced some structure into the neural network, such as intermediate spatial [47, 48, 49, 50, 51] and topological representations [52, 53, 54].”
BackgroundDriving Policy Transfer via Modularity and Abstraction
Müller, M., Dosovitskiy, A., Ghanem, B., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.09364 · Cited by 166 · cited 5×
“Modular learning can enable Sim-to-Real transfer [56, 61, 62]: one can shield learned modules from Sim-to-Real domain gaps by designing abstractions of the input raw sensor data that contain sufficient information to solve the task while being invariant to environmental factors that are hard to simulate accurately (li…”
BackgroundMethodInfluentialProcTHOR: Large-Scale Embodied AI Using Procedural Generation
Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Salvador, J., Ehsani, K., Han, W., Kolve, E., Farhadi, A., Kembhavi, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.06994 · Cited by 155 · cited 2×
“Looking at the top three teams on each challenge from 2022, four approaches can be characterized as pixel-to-action end to end (43, 45, 98, 99), only one can be characterized as modular (100), and one is unpublished.”
BackgroundMethod(2022) · doi:10.52202/068431-0433 · Cited by 155
Sim2Real in Robotics and Automation: Applications and Challenges
Hofer, S., Bekris, K., Handa, A., Gamboa, J. C., Mozifian, M., Golemo, F., Atkeson, C., Fox, D., Goldberg, K., Leonard, J., et al. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2021.3064065 · Cited by 154 · cited 4×
“Improving sim to close this gap would involve increasing RGB photo-realism or providing extensive plug-and-play RGB randomization, both hard open problems [88].”
BackgroundInfluentialPONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
Ramakrishnan, S. K., Chaplot, D. S., Al-Halah, Z., Malik, J., & Grauman, K. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01832 · Cited by 154 · cited 1×
“Representative examples of modular learning for exploration [61, 63] and for reaching object goals and image goals are [62, 64, 65, 66].”
BackgroundSelf-citationMulti-view deep learning for consistent semantic mapping with RGB-D cameras
Ma, L., Stuckler, J., Kerl, C., & Cremers, D. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202213 · Cited by 150 · cited 2×
“Semantic SLAM methods [19, 20, 21, 22, 23, 24] naturally extend SLAM to detect objects and keep them in memory in a spatial semantic map but offer no solution for efficient semantic exploration.”
MethodMapNet: An Allocentric Spatial Memory for Mapping Environments
Henriques, J. F., & Vedaldi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00884 · Cited by 146 · cited 1×
“While end-to-end policies often directly map sensor data to actions, akin to a modern version of Rodney Brook’s Subsumption architecture [46], researchers have also introduced some structure into the neural network, such as intermediate spatial [47, 48, 49, 50, 51] and topological representations [52, 53, 54].”
BackgroundSemantic SLAM Based on Object Detection and Improved Octomap
Zhang, L., Wei, L., Shen, P., Wei, W., Zhu, G., & Song, J. (2018) · IEEE Access · doi:10.1109/access.2018.2873617 · Cited by 144 · cited 2×
“Semantic SLAM methods [19, 20, 21, 22, 23, 24] naturally extend SLAM to detect objects and keep them in memory in a spatial semantic map but offer no solution for efficient semantic exploration.”
MethodCartman: The Low-Cost Cartesian Manipulator that Won the Amazon Robotics Challenge
Morrison, D., Tow, A. W., McTaggart, M., Smith, R., Kelly-Boxall, N., Wade-McCue, S., Erskine, J., Grinover, R., Gurman, A., Hunn, T., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8463191 · Cited by 142
Manhattan scene understanding using monocular, stereo, and 3D features
Flint, A., Murray, D., & Reid, I. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126501 · Cited by 132 · cited 2×
“Semantic SLAM meth-ods [19, 20, 21, 22, 23, 24, 25] naturally extend SLAM to detect objects and keep them in memory but offer no solution for efficient semantic exploration.”
BackgroundMethodGrounded Language Learning in a Simulated 3D World
Hermann, K. M., Hill, F., Green, S., Wang, F., Faulkner, R., Soyer, H., Szepesvári, D., Czarnecki, W. M., Jaderberg, M., Teplyashin, D., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06551 · Cited by 132 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundEnd-to-End Driving Via Conditional Imitation Learning
Codevilla, F., Muller, M., Lopez, A., Koltun, V., & Dosovitskiy, A. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460487 · Cited by 129
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
Savva, M., Chang, A. X., Dosovitskiy, A., Funkhouser, T. A., & Koltun, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.03931 · Cited by 127 · cited 2×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundGated-Attention Architectures for Task-Oriented Language Grounding
Chaplot, D. S., Mysore Sathyendra, K., Pasumarthi, R. K., Rajagopal, D., & Salakhutdinov, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11832 · Cited by 125
Self-citationAuxiliary Tasks and Exploration Enable ObjectGoal Navigation
Ye, J., Batra, D., Das, A., & Wijmans, E. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01581 · Cited by 120 · cited 2×
“First, the policy often detects the goal object but fails to stop nearby, which is consistent with prior work observing this ”last mile” failure in simulation [41].”
BackgroundResultSelf-citationHabitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
Ramakrishnan, S. K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A. X., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.08238 · Cited by 106 · cited 1×
“We replicate a single home in simulation as 3D scanning and digitizing a house — with the same Matterport camera used to digitize homes of the sim benchmark [94] — takes a few hours.”
MethodSelf-citationThe Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environments
Kemp, C. C., Edsinger, A., Clever, H. M., & Matulevich, B. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811922 · Cited by 94
Waypoint Models for Instruction-guided Navigation in Continuous Environments
Krantz, J., Gokaslan, A., Batra, D., Lee, S., & Maksymets, O. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01488 · Cited by 94
Self-citationHabitat-Matterport 3D Semantics Dataset
Yadav, K., Ramrakhya, R., Ramakrishnan, S. K., Gervet, T., Turner, J., Gokaslan, A., Maestre, N., Chang, A. X., Batra, D., Savva, M., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00477 · Cited by 92 · cited 2×
“We compare policies by Success Rate (SR) on a sim benchmark (single-floor navigation episodes of val split of the 2022 Habitat Challenge with 1093 episodes in 20 simulated homes of the HM3D Semantics dataset [92]), and the same home in sim and the real world (10 episodes).”
BackgroundMethodSelf-citationHabitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale
Ramrakhya, R., Undersander, E., Batra, D., & Das, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00511 · Cited by 90 · cited 5×
“Given our robot camera parameters are different from those used to train the original Habitat-Web policy — ( 640 x 480 ) frames with a 42 ◦ horizontal field of view, as opposed to ( 480 x 640 ) frames with a 79 ◦ horizontal field of view — we retrain the policy with robot camera parameters replicated in simulation.”
BackgroundInfluentialSelf-citationGated-Attention Architectures for Task-Oriented Language Grounding
Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., & Salakhutdinov, R. (2017) · arXiv · doi:10.48550/arxiv.1706.07230 · Cited by 82 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundSelf-citationVing: Learning open-world navigation with visual goals
Shah, D., Eysenbach, B., Kahn, G., Rhinehart, N., & Levine, S. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561936 · Cited by 81 · cited 1×
“The few other real-world semantic navigation works directly train on real-world images for outdoor visual goals [85, 86] and language instruction following [87].”
BackgroundFILM: Following Instructions in Language with Modular Methods
Min, S. Y., Chaplot, D. S., Ravikumar, P., Bisk, Y., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2110.07342 · Cited by 81 · cited 1×
“Modular learning methods have also been applied to longer-horizon tasks such as following language navigation instructions [67, 68], executing language instructions interactively in ALFRED [69, 70, 71], object rearrangement in AI2 Thor [72, 73] and improving perception using active exploration [74, 75].”
MethodInfluentialSelf-citationTHDA: Treasure Hunt Data Augmentation for Semantic Navigation
Maksymets, O., Cartillier, V., Gokaslan, A., Wijmans, E., Galuba, W., Lee, S., & Batra, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01509 · Cited by 65 · cited 1×
“Most relevant to us, [41, 42, 43] propose end-to-end policies to navigate to object goals.”
BackgroundSelf-citationSemantic Curiosity for Active Visual Learning
Chaplot, D. S., Jiang, H., Gupta, S., & Gupta, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_19 · Cited by 64
Self-citationTIDEE: Tidying Up Novel Rooms Using Visuo-Semantic Commonsense Priors
Sarch, G. H., Fang, Z., Harley, A. W., Schydlo, P., Tarr, M. J., Gupta, S., & Fragkiadaki, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19842-7_28 · Cited by 64
Bi-Directional Domain Adaptation for Sim2Real Transfer of Embodied Navigation Agents
Truong, J., Chernova, S., & Batra, D. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3062303 · Cited by 62 · cited 1×
“Does sim performance reflect real-world performance? Do design choices that improve sim performance improve real-world performance? Do sim error modes reflect real-world error modes? Most prior Sim-to-Real studies in navigation focused on spatial (point goal) navigation and legged locomotion [80, 81, 82, 83, 84], as o…”
BackgroundSelf-citationNavigating to Objects in Unseen Environments by Distance Prediction
Zhu, M., Zhao, B., & Kong, T. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981766 · Cited by 53
ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints
Shah, D., & Levine, S. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.019 · Cited by 52 · cited 1×
“The few other real-world semantic navigation works directly train on real-world images for outdoor visual goals [85, 86] and language instruction following [87].”
BackgroundSemi-parametric Topological Memory for Navigation
Savinov, N., Dosovitskiy, A., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00653 · Cited by 43 · cited 2×
“While end-to-end policies often directly map sensor data to actions, akin to a modern version of Rodney Brook’s Subsumption architecture [46], researchers have also introduced some structure into the neural network, such as intermediate spatial [47, 48, 49, 50, 51] and topological representations [52, 53, 54].”
BackgroundMurray, M. H., & Çakmak, M. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3178804 · Cited by 42 · cited 2×
“Modular learning methods have also been applied to longer-horizon tasks, such as following language navigation instructions (68, 69), executing language instructions interactively in ALFRED (Action Learning From Realistic Environments and Directives) (70–72), executing object rearrangement in AI2-Thor (73, 74), and im…”
MethodInfluentialRethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in NavigationPreprint
Truong, J., Rudolph, M., Yokoyama, N., Chernova, S., Batra, D., & Rai, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.10821 · Cited by 40 · cited 1×
“Does sim performance reflect real-world performance? Do design choices that improve sim performance improve real-world performance? Do sim error modes reflect real-world error modes? Most prior Sim-to-Real studies in navigation focused on spatial (point goal) navigation and legged locomotion [80, 81, 82, 83, 84], as o…”
BackgroundSelf-citationION: Instance-level Object Navigation
Li, W., Song, X., Bai, Y., Zhang, S., & Jiang, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475575 · Cited by 39 · cited 1×
“All methods we evaluate are straightforward to extend to a larger finite set of categories, and opportunities for future work include extensions to an unbounded set of object categories with open-vocabulary detectors [95] and instance-level object goals [96].”
MethodIs Mapping Necessary for Realistic PointGoal Navigation?
Partsey, R., Wijmans, E., Yokoyama, N., Dobosevych, O., Batra, D., & Maksymets, O. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01672 · Cited by 37
Self-citationBenchmarking Classic and Learned Navigation in Complex 3D Environments
Mishkin, D., Dosovitskiy, A., & Koltun, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10915 · Cited by 34 · cited 1×
“Learned navigation policies have predominantly been evaluated in sim [77].”
BackgroundOnline Learning of Reusable Abstract Models for Object Goal Navigation
Campari, T., Lamanna, L., Traverso, P., Serafini, L., & Ballan, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01445 · Cited by 26 · cited 1×
“Brook’s subsumption architecture (46), researchers have also introduced some structure into the neural network, such as intermediate spatial (47– 51) and topological representations (52–54) or a finite set machine of visual state abstractions (55).”
BackgroundVisual Semantic Navigation using Scene Priors
Yang, W. P., Wang, X., Farhadi, A., Gupta, A., & Mottaghi, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06543 · Cited by 25 · cited 1×
“While end-to-end policies often directly map sensor data to actions, akin to a modern version of Rodney Brook’s Subsumption architecture [46], researchers have also introduced some structure into the neural network, such as intermediate spatial [47, 48, 49, 50, 51] and topological representations [52, 53, 54].”
BackgroundArnold: An Autonomous Agent to Play FPS Games
Chaplot, D. S., & Lample, G. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10534 · Cited by 20 · cited 1×
“Inspired by seminal proofs of concept in autonomous driving [26, 27] and early successes of pixel-to-action deep reinforcement learning [28, 29], early applications of end-to-end learning to semantic navigation include [30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40].”
BackgroundSelf-citationA Simple Approach for Visual Rearrangement: 3D Mapping and Semantic SearchPreprint
Trabucco, B., Sigurdsson, G., Piramuthu, R., Sukhatme, G. S., & Salakhutdinov, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.13396 · Cited by 17 · cited 1×
“Modular learning methods have also been applied to longer-horizon tasks such as following language navigation instructions [67, 68], executing language instructions interactively in ALFRED [69, 70, 71], object rearrangement in AI2 Thor [72, 73] and improving perception using active exploration [74, 75].”
MethodDetecting Twenty-thousand Classes using Image-level SupervisionPreprint
Zhou, X., Girdhar, R., Joulin, A., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.02605 · Cited by 15 · cited 1×
“All methods we evaluate are straightforward to extend to a larger finite set of categories, and opportunities for future work include extensions to an unbounded set of object categories with open-vocabulary detectors [95] and instance-level object goals [96].”
MethodCoupling Vision and Proprioception for Navigation of Legged Robots
Fu, Z., Kumar, A., Agarwal, A., Qi, H., Malik, J., & Pathak, D. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00508 · Cited by 6
Self-citation1st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)
An, D., Wang, Z., Li, Y., Wang, Y., Hong, Y., Huang, Y., Wang, L., & Shao, J. (2022) · arXiv · doi:10.48550/arxiv.2206.11610 · Cited by 6 · cited 1×
“Modular learning methods have also been applied to longer-horizon tasks such as following language navigation instructions [67, 68], executing language instructions interactively in ALFRED [69, 70, 71], object rearrangement in AI2 Thor [72, 73] and improving perception using active exploration [74, 75].”
MethodInfluentialHabitat-Matterport 3D Semantics DatasetPreprint
Yadav, K., Ramrakhya, R., Ramakrishnan, S. K., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.05633 · cited 2×
“We compare policies by Success Rate (SR) on a sim benchmark (single-floor navigation episodes of val split of the 2022 Habitat Challenge with 1093 episodes in 20 simulated homes of the HM3D Semantics dataset [92]), and the same home in sim and the real world (10 episodes).”
BackgroundMethod
Research works printed but not linked: 16
- H. P. Moravec Obstacle avoidance and navigation in the real world by a seeing robot rover thesis Stanford University Palo Alto CA (1980).
- S. Thrun Robotic mapping: A survey in Exploring Artificial Intelligence in the New Millennium (Morgan Kaufmann Publishers Inc. 2002) pp. 1–35.
- U. Muller J. Ben E. Cosatto B. Flepp Y. Cun Off-road obstacle avoidance through end-to-end learning in Advances in Neural Information Processing Systems (MIT Press 2005) vol. 18.
- P. Mirowski R. Pascanu F. Viola H. Soyer A. Ballard A. Banino M. Denil R. Goroshin L. Sifre K. Kavukcuoglu D. Kumaran R. Hadsell Learning to navigate in complex environments ” paper presented at the 5th International Conference on Learning Representations Toulon France 24 to 26 2017 (ICLR 2017).
- A. Dosovitskiy V. Koltun “Learning to act by predicting the future ” paper presented at the 5th International Conference on Learning Representations Toulon France 24 to 26 2017 (ICLR 2017).
- P. Mirowski M. K. Grimes M. Malinowski K. M. Hermann K. Anderson D. Teplyashin K. Simonyan K. Kavukcuoglu A. Zisserman R. Hadsell Learning to navigate in cities without a map in Advances in Neural Information Processing Systems (Curran Associates Inc. 2018) vol. 31 pp. 2419–2430.
- E. Parisotto R. Salakhutdinov Neural map: Structured memory for deep reinforcement learning in International Conference on Learning Representations (ICLR) (2018).
- D. S. Chaplot E. Parisotto R. Salakhutdinov Active neural localization in International Conference on Learning Representations (ICLR) (2018).
- D. S. Chaplot D. P. Gandhi A. Gupta R. R. Salakhutdinov Object goal navigation using goal-oriented semantic exploration in Advances in Neural Information Processing Systems (Curran Associates Inc. 2020) vol. 33 p. 4247.
- M. Hahn D. S. Chaplot S. Tulsiani M. Mukadam J. M. Rehg A. Gupta No RL no simulation: Learning to navigate without navigating in Advances in Neural Information Processing Systems (Curran Associates Inc. 2021) vol. 34.
- X. Liu, H. Palacios, C. Muise, A planning based neural-symbolic approach for embodied instruction following. Interactions 9, 17 (2022).
- D. S. Chaplot M. Dalal S. Gupta J. Malik R. Salakhutdinov SEAL: Self-supervised Embodied Active Learning using exploration and 3D consistency in Advances in Neural Information Processing Systems (Curran Associates Inc. 2021).
- E. Kolve R. Mottaghi W. Han E. V. Bilt L. Weihs A. Herrasti M. Deitke K. Ehsani D. Gordon Y. Zhu A. Kembhavi A. Gupta A. Farhadi AI2-THOR: An interactive 3D environment for visual AI. arXiv:1712.05474 [cs.CV] (26 August 2022). · arXiv
- D. Kalashnikov A. Irpan P. Pastor J. Ibarz A. Herzog E. Jang D. Quillen E. Holly M. Kalakrishnan V. Vanhoucke S. Levine “Scalable deep reinforcement learning for vision-based robotic manipulation ” paper presented at the 2nd Annual Conference on Robot Learning (CoRL 2018) Zürich Switzerland 29 to 31 October 2018 pp. 651–673.
- A. Szot A. Clegg E. Undersander E. Wijmans Y. Zhao J. Turner N. Maestre M. Mukadam D. Chaplot O. Maksymets A. Gokaslan V. Vondrus S. Dharur F. Meier W. Galuba A. Chang Z. Kira V. Koltun J. Malik M. Savva D. Batra Habitat 2.0: Training home assistants to rearrange their habitat in Advances in Neural Information Processing Systems (Curran Associates Inc. 2021) vol. 34 p. 251.
- A. Murali T. Chen K. V. Alwala D. Gandhi L. Pinto S. Gupta A. Gupta PyRobot: An open-source robotics framework for research and benchmarking. arXiv:1906.08236 [cs.RO] (19 June 2019). · arXiv
Cited by
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., & Xia, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01370 · Cited by 281
“When applying these spatial concepts to reasoning, prior works often focus on explicit spatial scene memories [26, 27] or spatial scene graphs [29, 30, 59, 60].”
MethodDeep Reinforcement Learning for Robotics: A Survey of Real-World Successes
Tang, C., Abbatematteo, B., Hu, J., Chandra, R., Martín-Martín, R., & Stone, P. (2025) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-030323-022510 · Cited by 177
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., & Xia, F. (2024) · arXiv · doi:10.48550/arxiv.2401.12168 · Cited by 269
“When applying these spatial concepts to reasoning, prior works often focus on explicit spatial scene memories [26, 27] or spatial scene graphs [29, 30, 59, 60].”
MethodFoundation models in robotics: Applications, challenges, and the future
Firoozi, R., Tucker, J., Tian, S., Majumdar, A., Sun, J., Liu, W., Zhu, Y., Song, S., Kapoor, A., Hausman, K., et al. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241281508 · Cited by 238
“Semantic navigation [142] seeks to address this question by presenting an empirical analysis of semantic visual navigation methods.”
NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration
Sridhar, A., Shah, D., Glossop, C., & Levine, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610665 · Cited by 471
“However, policies trained in simulation tend to transfer poorly to real-world environments [16, 22], and our experiments reveal that even the best exploration policies trained on real-world data perform poorly in complex indoor and outdoor environments.”
Background
Metadata
- Authors
- Gervet, T., Chintala, S., Batra, D., Malik, J., & Chaplot, D. S.
- Affiliations
- Carnegie Mellon University, Pittsburgh, PA, USA.
- Meta (United States) (US)
- Georgia Institute of Technology (US)
- Venue
- Science Robotics
- Published
- 2023-06-28
- Volume
- 8
- Issue
- 79
- Pages
- eadf6991-eadf6991
- ISSN
- 2470-9476 (ISSN-L)
- OpenAlex
- W4382362889
- Cited by
- 125
- References
- 83
- Citations by year
- 2026: 17 · 2025: 50 · 2024: 44 · 2023: 14
- FWCI
- 11.06
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Science Robotics (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- mobile robots, object navigation, sim-to-real transfer, modular learning, end-to-end learning, exploration, domain gap, family environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.