Learning to Set Waypoints for Audio-Visual Navigation
Open accessAbstract
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e.g., a phone ringing in another room). Existing models learn to act at a fixed granularity of agent motion and rely on simple recurrent aggregations of the audio observations. We introduce a reinforcement learning approach to audio-visual navigation with two key novel elements: 1) waypoints that are dynamically set and learned end-to-end within the navigation policy, and 2) an acoustic memory that provides a structured, spatially grounded record of what the agent has heard as it moves. Both new ideas capitalize on the synergy of audio and visual data for revealing the geometry of an unmapped space. We demonstrate our approach on two challenging datasets of real-world 3D scenes, Replica and Matterport3D. Our model improves the state of the art by a substantial margin, and our experiments reveal that learning the links between sights, sounds, and space is essential for audio-visual navigation. Project: http://vision.cs.utexas.edu/projects/audio_visual_waypoints.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Under the Depth setting in the Unheard sound task on Replica, AGVP achieves relative improvements of 34.8% in SPL, 25.9% in SR, and 7.4% in SNA over AV-WaN.
“under the Depth setting in the Unheard sound task, AGVP achieves relative improvements of 34.8% in SPL, 25.9% in SR, and 7.4% in SNA over AV-WaN [3].”
From Audio-Guided Visual Perception for Audio-Visual Navigation (2025) · cites this paper as [3]
Table 1 (4 Experiments, p. 8) reports AV-WaN’s SPL, SR, and SNA on Replica with unheard sounds, providing a baseline for comparison. It does not report AGVP results or the claimed 34.8%, 25.9%, and 7.4% relative improvements under a Depth setting. (LLM, conf=0.84)
Experimental — accuracy not yet measuredIn the Unheard sound task on Replica, AGVP surpasses AV-WaN across all metrics.
“In the most challenging Unheard sound task, AGVP surpasses AVWaN [3] across all metrics.”
From Audio-Guided Visual Perception for Audio-Visual Navigation (2025) · cites this paper as [3]
In the Unheard sound task under the Depth setting on Matterport3D, AGVP achieves a success rate of 60.8%, a 7.2% relative improvement over AV-WaN, with better path efficiency.
“In the Unheard sound task under the Depth setting, AGVP achieves a success rate of 60.8%, a 7.2% relative improvement over AV-WaN [3], while also exhibiting better path efficiency.”
From Audio-Guided Visual Perception for Audio-Visual Navigation (2025) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Thrun, S. (2002) · CACM · doi:10.1145/504729.504754 · Cited by 7,941 · see citing works
Matterport3D: Learning from RGB-D Data in Indoor Environments
Chang, A., Dai, A., Funkhouser, T., Halber, M., Niebner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00081 · Cited by 1,858 · see citing works
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · see citing works
Visual simultaneous localization and mapping: a survey
Fuentes-Pacheco, J., Ruiz-Ascencio, J., & Rendón-Mancha, J. M. (2015) · Artificial Intelligence Review · doi:10.1007/s10462-012-9365-8 · Cited by 883 · see citing works
A Recurrent Latent Variable Model for Sequential DataPreprint
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02216 · Cited by 799 · see citing works
The Replica Dataset: A Digital Replica of Indoor SpacesPreprint
Straub, J., Whelan, T. J., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J. J., Mur-Artal, R., Ren, C. Y., Verma, S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05797 · Cited by 711 · see citing works
Learning to Navigate in Complex EnvironmentsPreprint
Mirowski, P., Pascanu, R., Viola, F., Soyer, H., Ballard, A. J., Banino, A., Denil, M., Goroshin, R., Sifre, L., Kavukcuoglu, K., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03673 · Cited by 679 · see citing works
Robust reconstruction of indoor scenes
Choi, S., Zhou, Q.-Y., & Koltun, V. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299195 · Cited by 634 · see citing works
On evaluation of embodied navigation agents
(2018) · doi:10.48550/arxiv.1807.06757 · Cited by 622 · see citing works
Early-blind human subjects localize sound sources better than sighted subjects
Lessard, N., Paré, M., Leporé, F., & Lassonde, M. C. (1998) · Nature · doi:10.1038/26228 · Cited by 599 · see citing works
Cognitive Mapping and Planning for Visual Navigation
Gupta, S., Davidson, J., Levine, S., Sukthankar, R., & Malik, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.769 · Cited by 521 · see citing works
Audio-Visual Event Localization in Unconstrained Videos
Tian, Y., Shi, J., Li, B., Duan, Z., & Xu, C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_16 · Cited by 506 · see citing works
Arandjelović, R., & Zisserman, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_27 · Cited by 448 · see citing works
Data-efficient hierarchical reinforcement learning
Nachum, O., Gu, S., Lee, H., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.08296 · Cited by 446 · see citing works
Gougoux, F., Zatorre, R. J., Lassonde, M. C., Voss, P., & Leporé, F. (2005) · PLoS Biology · doi:10.1371/journal.pbio.0030027 · Cited by 392 · see citing works
Learning to Localize Sound Source in Visual Scenes
Senocak, A., Oh, T.-H., Kim, J., Yang, M.-H., & Kweon, I. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00458 · Cited by 328 · see citing works
Learning to explore using active neural SLAM
Chaplot, D. S., Gandhi, D., Gupta, S., Gupta, A., & Salakhutdinov, R. (2020) · arXiv · doi:10.48550/arxiv.2004.05155 · Cited by 318 · see citing works
Localization of sound sources in robotics: A review
Rascón, C., & Meza, I. (2017) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2017.07.011 · Cited by 273 · see citing works
Acoustic echoes reveal room shape
Dokmanić, I., Parhizkar, R., Walther, A., et al. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1221464110 · Cited by 253 · see citing works
Neural Topological SLAM for Visual Navigation
Singh Chaplot, D., Salakhutdinov, R., Gupta, A., & Gupta, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01289 · Cited by 241 · see citing works
Audio Vision: Using Audio-Visual Synchrony to Locate Sounds
Hershey, J. R., & Movellan, J. R. (1999) · Cited by 227 · see citing works
Nakadai, K., Lourens, T., Okuno, H. G., & Kitano, H. (2000) · Cited by 224 · see citing works
The Option-Critic Architecture
Bacon, P., Harb, J., & Precup, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10916 · Cited by 219 · see citing works
SoundSpaces: Audio-Visual Navigation in 3D Environments
Chen, C., Jain, U., Schissler, C., Gari, S. V. A., Al-Halah, Z., Ithapu, V. K., Robinson, P., & Grauman, K. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_2 · Cited by 202 · see citing works
Self-citationSemi-parametric Topological Memory for NavigationPreprint
Savinov, N., Dosovitskiy, A., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00653 · Cited by 194 · see citing works
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
Fang, K., Toshev, A., Fei-Fei, L., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00063 · Cited by 178 · see citing works
Driving Policy Transfer via Modularity and AbstractionPreprint
Müller, M., Dosovitskiy, A., Ghanem, B., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.09364 · Cited by 167 · see citing works
MapNet: An Allocentric Spatial Memory for Mapping Environments
Henriques, J. F., & Vedaldi, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00884 · Cited by 148 · see citing works
Learning Multi-Level Hierarchies with HindsightPreprint
Levy, A., Konidaris, G., Platt, R., & Saenko, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00948 · Cited by 144 · see citing works
Combining Optimal Control and Learning for Visual Navigation in Novel EnvironmentsPreprint
Bansal, S., Tolani, V., Gupta, S., Malik, J., & Tomlin, C. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.02531 · Cited by 141 · see citing works
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
Gan, C., Zhang, Y., Wu, J., Gong, B., & Tenenbaum, J. B. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197008 · Cited by 124 · see citing works
Ego-Topo: Environment Affordances From Egocentric Video
Nagarajan, T., Li, Y., Feichtenhofer, C., & Grauman, K. L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00024 · Cited by 100 · see citing works
Self-citationVisualEchoes: Spatial Image Representation Learning Through Echolocation
Gao, R., Chen, C., Al-Halah, Z., Schissler, C., & Grauman, K. L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_38 · Cited by 79 · see citing works
Self-citationSpatial Action Maps for Mobile Manipulation
Wu, J., Sun, X., Zeng, A., Song, S., Lee, J., Rusinkiewicz, S. M., & Funkhouser, T. A. (2020) · Robotics: Science and Systems · doi:10.15607/rss.2020.xvi.035 · Cited by 69 · see citing works
Unifying Map and Landmark Based Representations for Visual NavigationPreprint
Gupta, S., Fouhey, D. F., Levine, S., & Malik, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.08125 · Cited by 62 · see citing works
Benchmarking Classic and Learned Navigation in Complex 3D EnvironmentsPreprint
Mishkin, D., Dosovitskiy, A., & Koltun, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10915 · Cited by 57 · see citing works
BatVision: Learning to See 3D Spatial Layout with Two Ears
Christensen, J. H., Hornauer, S., & Yu, S. X. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196934 · Cited by 56 · see citing works
Learning Spatial Common Sense With Geometry-Aware Recurrent Networks
Tung, H.-Y. F., Cheng, R., & Fragkiadaki, K. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00270 · Cited by 54 · see citing works
(Sasha), A. S., Emi, B., Zamir, A., Guibas, L. J., Savarese, S., & Malik, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.11971 · Cited by 51 · see citing works
Learning over Subgoals for Efficient Navigation of Structured, Unknown Environments.
Stein, G., Bradley, C., & Roy, N. (2018) · Conference on Robot Learning · Cited by 32 · see citing works
InfoBot: Transfer and Exploration via the Information BottleneckPreprint
Goyal, A., Islam, R., Strouse, D., Ahmed, Z., Larochelle, H., Botvinick, M., Bengio, Y., & Levine, S. (2019) · arXiv (Cornell University) · Cited by 28 · see citing works
Deep learning of structured environments for robot search
Caley, J. A., Lawrance, N., & Hollinger, G. A. (2019) · Autonomous Robots · doi:10.1007/s10514-018-09821-4 · Cited by 26 · see citing works
Deep learning of structured environments for robot search
Caley, J. A., Lawrance, N., & Hollinger, G. A. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759587 · Cited by 25 · see citing works
Sound localization and multi-modal steering for autonomous virtual agents
Wang, Y., Kapadia, M., Huang, P., Kavan, L., & Badler, N. I. (2014) · ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D) · doi:10.1145/2556700.2556718 · Cited by 15 · see citing works
Sound Source Localization and Separation
Nakadai, K., & Nakamura, K. (2015) · Wiley Encyclopedia of Electrical and Electronics Engineering · doi:10.1002/047134608x.w8266 · Cited by 14 · see citing works
BatVision: Learning to See 3D Spatial Layout with Two EarsPreprint
Christensen, J. H., Hornauer, S., & Yu, S. X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.07011 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In International Conference on Learning Representations , 2019. URL https://openreview. net/pdf?id=SyMWn05F7 .
- Chengshu Li, Fei Xia, Roberto Martin-Martin, and Silvio Savarese. Hrl4in: Hierarchical reinforcement learning for interactive navigation with mobile manipulators. In CoRL , 2019.
- Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. In ICLR , 2020.
Cited by
Sorted by citations
A Metaverse: Taxonomy, Components, Applications, and Open Challenges
Park, S., & Kim, Y. (2022) · IEEE Access · doi:10.1109/access.2021.3140175 · Cited by 1,926 · see citing works
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
Ramakrishnan, S. K., Chaplot, D. S., Al-Halah, Z., Malik, J., & Grauman, K. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01832 · Cited by 154 · see citing works
Self-citationEvaluation of Socially-Aware Robot Navigation
Gao, Y., & Huang, C. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.721317 · Cited by 114 · see citing works
Hong, Y., Wang, Z., Wu, Q., & Gould, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01500 · Cited by 94 · see citing works
Semantic Audio-Visual Navigation
Chen, C., Al-Halah, Z., & Grauman, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01526 · Cited by 96 · see citing works
Self-citationWaypoint Models for Instruction-guided Navigation in Continuous Environments
Krantz, J., Gokaslan, A., Batra, D., Lee, S., & Maksymets, O. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01488 · Cited by 94 · see citing works
Toward Practical Monocular Indoor Depth Estimation
Wu, C.-Y., Wang, J., Hall, M., Neumann, U., & Su, S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00379 · Cited by 75 · see citing works
Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation
Al-Halah, Z., Ramakrishnan, S. K., & Grauman, K. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01652 · Cited by 58 · see citing works
Self-citationAudio-Visual Grouping Network for Sound Localization from Mixtures
Mo, S., & Tian, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01018 · Cited by 48 · see citing works
Motion Planning and Cooperative Manipulation for Mobile Robots With Dual Arms
Sun, F., Chen, Y., Wu, Y., Li, L., & Ren, X. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3146387 · Cited by 42 · see citing works
Layout-based Causal Inference for Object Navigation
Zhang, S., Song, X., Li, W., Bai, Y., Yu, X., & Jiang, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01039 · Cited by 41 · see citing works
Core Challenges in Embodied Vision-Language Planning
Francis, J., Kitamura, N., Labelle, F., Lu, X., Navarro, I., & Oh, J. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13646 · Cited by 40 · see citing works
Move2Hear: Active Audio-Visual Source Separation
Majumder, S., Al-Halah, Z., & Grauman, K. L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00034 · Cited by 40 · see citing works
Self-citationAudio-Visual Class-Incremental Learning
Pian, W., Mo, S., Guo, Y., & Tian, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00717 · Cited by 39 · see citing works
RobustNav: Towards Benchmarking Robustness in Embodied Navigation
Chattopadhyay, P., Hoffman, J., Mottaghi, R., & Kembhavi, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01540 · Cited by 37 · see citing works
Catch Me if You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments With Moving Sounds
Younes, A., Honerkamp, D., Welschehold, T., & Valada, A. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3234766 · Cited by 35 · see citing works
Audio-Visual Floorplan Reconstruction
Purushwalkam, S., Garí, S. V. A., Ithapu, V. K., Schissler, C., Robinson, P. W., Gupta, A., & Grauman, K. L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00122 · Cited by 34 · see citing works
Self-citationKästner, L., Zhao, X., Buiyan, T., Li, J., Shen, Z., Lambrecht, J., & Marx, C. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636039 · Cited by 29 · see citing works
Mo, S., & Morgado, P. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02567 · Cited by 29 · see citing works
Transformer Memory for Interactive Visual Navigation in Cluttered Environments
Li, W., Hong, R., Shen, J., Yuan, L., & Lu, Y. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3241803 · Cited by 25 · see citing works
Kästner, L., Fatloun, B., Shen, Z., Gawrisch, D., & Lambrecht, J. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812111 · Cited by 24 · see citing works
Hyperbolic Audio-visual Zero-shot Learning
Hong, J., Hayder, Z., Han, J., Fang, P., Harandi, M. T., & Petersson, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00724 · Cited by 22 · see citing works
Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulation
Schmalstieg, F., Honerkamp, D., Welschehold, T., & Valada, A. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3329619 · Cited by 21 · see citing works
Echo-Enhanced Embodied Visual Navigation
Yu, Y., Cao, L., Sun, F., Yang, C., Lai, H., & Huang, W. (2023) · Neural Computation · doi:10.1162/neco_a_01579 · Cited by 18 · see citing works
Finding Fallen Objects Via Asynchronous Audio-Visual Integration
Gan, C., Gu, Y., Zhou, S., Schwartz, J., Alter, S., Traer, J. A., Gutfreund, D., Tenenbaum, J. B., McDermott, J. H., & Torralba, A. B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01027 · Cited by 15 · see citing works
Deep Learning for Embodied Vision Navigation: A SurveyPreprint
Zhu, F., Zhu, Y., Lee, V. C., Liang, X., & Chang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04097 · Cited by 19 · see citing works
Wang, H., Wang, Y., Zhong, F., Wu, M., Zhang, J., Wang, Y., & Dong, H. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3272518 · Cited by 14 · see citing works
Learning-Augmented Model-Based Planning for Visual Exploration
Li, Y., Debnath, A., Stein, G. J., & Košecká, J. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341773 · Cited by 12 · see citing works
Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigation
Chen, J., Wang, W., Liu, S., Li, H., & Yang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01009 · Cited by 12 · see citing works
REALIMPACT: A Dataset of Impact Sound Fields for Real Objects
Clarke, S., Gao, R., Wang, M. L., Rau, M., Xu, J., Wang, J.-H., James, D. L., & Wu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00152 · Cited by 11 · see citing works
Self-citationSymmetry-aware Neural Architecture for Embodied Visual Navigation
Liu, S., Suganuma, M., & Okatani, T. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01909-4 · Cited by 11 · see citing works
Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation
Mejia, J., Dean, V., Hellebrekers, T., & Gupta, A. (2024) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra57147.2024.10611305 · Cited by 32 · see citing works
The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective
Jia, W., Liu, M., Jiang, H., Ananthabhotla, I., Rehg, J. M., Ithapu, V. K., & Gao, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02493 · Cited by 11 · see citing works
Self-citationMeasuring Acoustics with Collaborative Multiple Agents
Yu, Y., Chen, C., Cao, L., Yang, F., & Sun, F. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/38 · Cited by 10 · see citing works
Self-citationBinaural SoundNet: Predicting Semantics, Depth and Motion With Binaural Sounds
Dai, D., Vasudevan, A. B., Matas, J., & Gool, L. V. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3155643 · Cited by 10 · see citing works
Sun, L., Kanezaki, A., Caron, G., & Yoshiyasu, Y. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103135 · Cited by 9 · see citing works
Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear
Gao, R., Li, H., Dharan, G., Wang, Z., Li, C., Xia, F., Savarese, S., Fei-Fei, L., & Wu, J. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160461 · Cited by 9 · see citing works
Self-citationAdVerb: Visually Guided Audio Dereverberation
Chowdhury, S., Ghosh, S., Dasgupta, S., Ratnarajah, A. J., Tyagi, U., & Manocha, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00725 · Cited by 9 · see citing works
Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations
Majumder, S., Jiang, H., Moulon, P., Henderson, E., Calamia, P. T., Grauman, K. L., & Ithapu, V. K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01017 · Cited by 8 · see citing works
Self-citationSound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation
Chen, Z., Qian, S., & Owens, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00726 · Cited by 9 · see citing works
Deep Cross-Modal Retrieval Between Spatial Image and Acoustic Speech
Qian, X., Xue, W., Zhang, Q., Tao, R., & Li, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3323876 · Cited by 7 · see citing works
Multi-Goal Audio-Visual Navigation Using Sound Direction Map
Kondoh, H., & Kanezaki, A. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341819 · Cited by 7 · see citing works
RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation
Yang, Z., Lin, J., Chen, P., Cherian, A., Marks, T. K., Roux, J. L., & Gan, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01538 · Cited by 6 · see citing works
Rao, P. P., & Chowdhury, A. R. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812193 · Cited by 6 · see citing works
CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
Liu, X., Paul, S., Chatterjee, M., & Cherian, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i4.28167 · Cited by 6 · see citing works
Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction
Chen, C., Ramos, J., Tomar, A., & Grauman, K. L. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802636 · Cited by 4 · see citing works
Self-citationLearning to Map Efficiently by Active Echolocation
Hu, X., Purushwalkam, S., Harwath, D. F., & Grauman, K. L. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341664 · Cited by 2 · see citing works
Self-citationAudio Inputs for Active Speaker Detection and Localization Via Microphone Array
Berghi, D., & Jackson, P. J. B. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248185 · Cited by 2 · see citing works
Geometry-aware multi-task learning for binaural audio generation from videoPreprint
Garg, R. (2021) · arXiv (Cornell University) · doi:10.26153/tsw/14434 · Cited by 2 · see citing works
Map-Guided Few-Shot Audio-Visual Acoustics Modeling
Huang, D.-W., Lin, K., Chen, P., & Du, Q. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889097 · Cited by 1 · see citing works
Disentangled Acoustic Fields For Multimodal Physical Scene Understanding
Yin, J., Luo, A., Du, Y., Cherian, A., Marks, T. K., Roux, J. L., & Gan, C. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802429 · Cited by 1 · see citing works
Audio-Guided Visual Perception for Audio-Visual Navigation
Wang, Y., Yu, Y., Sun, F., Wang, L., & Zheng, W. (2025) · 2025 International Conference on Virtual Reality and Visualization (ICVRV) · doi:10.1109/icvrv67992.2025.00184
VSGM -- Enhance robot task understanding ability through visual semantic graphPreprint
Tsai, C. Y., & Su, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08959
Imagination-Augmented Cost Map for Local Navigation
Shen, Z., Kästner, L., Yordanova, M., & Lambrecht, J. (2023) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/ur57808.2023.10202542
HomeEmergency - Using Audio to Find and Respond to Emergencies in the Home
Mullen, J. F., Kumar, D., Qi, X., Madhivanan, R., Sen, A., Manocha, D., & Kim, R. Y. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3561570
- Authors
- Chen, C., Majumder, S., Al-Halah, Z., Gao, R., Ramakrishnan, S. K., & Grauman, K.
- Affiliations
- The University of Texas at Austin (US)
- Stanford University (US)
- Meta (United Kingdom) (GB)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2008.09622
- OpenAlex
- W3121780787
- Semantic Scholar
- 102d3e153bb402a8533af2ee03cb0ba536d09f2f
- Cited by
- 53
- References
- 49
- Citations by year
- 2025: 3 · 2024: 6 · 2023: 23 · 2022: 12 · 2021: 9
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
- Keywords
- auditory memory, sound localization, reinforcement learning, 3D navigation, spatial grounding, multisensory perception, Matterport3D
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
