A real 3d embodied dataset for robotic active visual learning
Abstract
Active interaction with environments is one of the striking characteristics of robotic active vision, which allows robots to move to facilitate visual tasks. Recently, several embodied AI platforms have been proposed as the synthetic environments to study robotic active vision, without needing to interact in real world. However, by using synthetic data, model trained on these platforms will suffer performance degradation when applied in reality. In this letter, a real 3D embodied dataset is proposed for robotic active visual learning. The proposed dataset consists of real point cloud data densely collected in 7 real-world indoor scenes. In our embodied dataset, researchers are able to simulate the movements and interactions of robots in indoor environments and obtain real visual data, which will not lead to performance degradation in reality. Furthermore, we proposed a 3D divergency policy that can guide robots to move and collect data to improve visual performance in novel environments. The proposed policy is designed following a simple fact: a good 3D detector should produce consistent 3D detection results for the same object from different viewpoints. Therefore, our policy encourages the robot to explore the area where the detector generates different 3D bounding boxes for the same object and helps the robot improve its visual performance in novel scenes.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AutoAlignV2 enhances cross-modal feature fusion through deformable feature alignment, making it more effective in small-object detection such as pedestrians and bicycles.
“AutoAlignV2 [259] enhances cross-modal feature fusion through deformable feature alignment, making it more effective in small-object detection (e.g., pedestrians and bicycles).”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [259]
Cites
The Pascal Visual Object Classes Challenge: A Retrospective
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0733-5 · Cited by 7,139 · see citing works
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · see citing works
SUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · see citing works
Deep Hough Voting for 3D Object Detection in Point Clouds
Qi, C. R., Litany, O., He, K., & Guibas, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00937 · Cited by 1,244 · see citing works
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · see citing works
Aloimonos, J., Weiss, I., & Bandyopadhyay, A. (1988) · International Journal of Computer Vision · doi:10.1007/bf00133571 · Cited by 931 · see citing works
Gibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763 · see citing works
The Replica Dataset: A Digital Replica of Indoor SpacesPreprint
Straub, J., Whelan, T. J., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J. J., Mur-Artal, R., Ren, C. Y., Verma, S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05797 · Cited by 711 · see citing works
Niroui, F., Zhang, K., Kashino, Z., & Nejat, G. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2891991 · Cited by 389 · see citing works
Bajcsy, R. K., Aloimonos, Y., & Tsotsos, J. K. (2017) · Autonomous Robots · doi:10.1007/s10514-017-9615-3 · Cited by 338 · see citing works
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
Deitke, M., Han, W., Herrasti, A., Kembhavi, A., Kolve, E., Mottaghi, R., Salvador, J., Schwenk, D., VanderBilt, E., Wallingford, M., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00323 · Cited by 220 · see citing works
A dataset for developing and benchmarking active vision
Ammirato, P., Poirson, P., Park, E., Kosecka, J., & Berg, A. C. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989164 · Cited by 187 · see citing works
Tianshou: a Highly Modularized Deep Reinforcement Learning LibraryPreprint
Weng, J., Chen, H., Yan, D., You, K., Duburcq, A., Zhang, M., Su, Y., Su, H., & Zhu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.14171 · Cited by 137 · see citing works
Sim2Real Viewpoint Invariant Visual Servoing by Recurrent Control
Sadeghi, F., Toshev, A., Jang, E., & Levine, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00493 · Cited by 133 · see citing works
An Exploration of Embodied Visual Exploration
Ramakrishnan, S. K., Jayaraman, D., & Grauman, K. L. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01437-z · Cited by 101 · see citing works
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks
Jayaraman, D., & Grauman, K. L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00135 · Cited by 87 · see citing works
Multipart Vehicle Detection Using Symmetry-Derived Analysis and Active Learning
Satzoda, R. K., & Trivedi, M. M. (2015) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2015.2494586 · Cited by 69 · see citing works
Semantic Curiosity for Active Visual Learning
Chaplot, D. S., Jiang, H., Gupta, S., & Gupta, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_19 · Cited by 64 · see citing works
The Importance of Metric Learning for Robotic Vision: Open Set Recognition and Active Learning
Meyer, B. J., & Drummond, T. W. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794188 · Cited by 29 · see citing works
Active learning guided interactions for consistent image segmentation with reduced user interactions
Veeraraghavan, H., & Miller, J. V. (2011) · IEEE International Symposium on Biomedical Imaging: From Nano to Macro (ISBI) · doi:10.1109/isbi.2011.5872719 · Cited by 26 · see citing works
Active 6D Multi-Object Pose Estimation in Cluttered Scenarios with Deep Reinforcement Learning
Sock, J., Garcia-Hernando, G., & Kim, T. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340842 · Cited by 20 · see citing works
Adversarial Training on Point Clouds for Sim-to-Real 3D Object Detection
DeBortoli, R., Li, F., Kapoor, A., & Hollinger, G. A. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3093869 · Cited by 19 · see citing works
Move to See Better: Self-Improving Embodied Object DetectionPreprint
Fang, Z., Jain, A., Sarch, G. H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00057 · Cited by 14 · see citing works
Lobos-Tsunekawa, K., & Harada, T. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341771 · Cited by 11 · see citing works
Embodied Visual Active Learning for Semantic Segmentation
Nilsson, D., Pirinen, A., Gärtner, E., & Sminchisescu, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16338 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“Building upon Matterport3D [26], Room-to-Room (R2R) [261] was the first dataset to integrate natural language modalities, introducing the Vision-and-Language Navigation (VLN) task.”
Influential
- Authors
- Zhao, Q., Zhang, L., Wu, L., Qiao, H., & Liu, Z.
- Affiliations
- Chinese Academy of Sciences (CN)
- University of California, Santa Barbara (US)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2022-07
- Volume
- 7
- Issue
- 3
- Pages
- 6646-6652
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4226036061
- Cited by
- 17
- References
- 25
- Citations by year
- 2026: 1 · 2025: 12 · 2024: 1 · 2023: 2 · 2022: 1
- FWCI
- 1.61
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- Funders
- National Key Research and Development Plan of China (2020AAA0108902)
- Chinese Academy of Science (XDB32050100)
- NSFC, China (61627808)
- Dongguan Core Technology Research Frontier Project (2019622101001)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Keywords
- embodied AI, 3D object detection, robot exploration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
