A dataset for developing and benchmarking active vision
The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20,000+ RGB-D images and 50,000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the dataset for simulating active vision, and use the dataset to develop and evaluate a deep-network-based system for next best move prediction for object classification using reinforcement learning. Our dataset is available for download at cs.unc.edu/~ammirato/active_vision_dataset_website/.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The latter is the only unsupervised method, while the former three need training data to learn the policy.”
From Unsupervised Active Visual Search With Monte Carlo Planning Under Uncertain Detections (2024) · cites this paper as [9]
In the citing paper
“From results reported in Table 1 we can clearly see that our approach POMP-BE-PD outperforms EAT in terms of SR, with a little increment in APL, which is reasonable since we are now considering more challenging situations, as we will deeply explore in Sect. 4.3.”
From Unsupervised Active Visual Search With Monte Carlo Planning Under Uncertain Detections (2024) · cites this paper as [9]
In the citing paper
“With respect to the standard protocol defined in the benchmark paper [8], this protocol provides results only using GT annotations for object detection, and on a limited number of scenes. Moreover, it uses only a subset of target objects.”
From Unsupervised Active Visual Search With Monte Carlo Planning Under Uncertain Detections (2024) · cites this paper as [9]
In the citing paper
“Our motion policy explicitly exploits the knowledge of the environment for the visibility modelling, instead other RL-based strategies [9], [10] implicitly encode such environment knowledge in a data-driven manner.”
From Unsupervised Active Visual Search With Monte Carlo Planning Under Uncertain Detections (2024) · cites this paper as [9]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
(2016) · Cited by 31,590
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · cited 1×
“The datasets that have been a driving force in pushing the deep learning revolution in object recognition, Pascal VOC [4], the ImageNet Challenge [5], and MS COCO [6] are all collected from web images (usually from Flickr) using web search based on keywords.”
MethodStructure-from-Motion Revisited
Schonberger, J. L., & Frahm, J.-M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.445 · Cited by 6,433 · cited 2×
“For each scan of each scene, we create a sparse reconstruction of the scene using the RGB structure from motion tool COLMAP from Sch¨onberger et al [30], [31].”
MethodSelective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
3D ShapeNets: A deep representation for volumetric shapes
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298801 · Cited by 4,550
Multi-view Convolutional Neural Networks for 3D Shape Recognition
Su, H., Maji, S., Kalogerakis, E., & Learned-Miller, E. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.114 · Cited by 3,480
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
SUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283 · cited 1×
“We instead use the provided object masks to crop the object and overlay it on a random background sampled from SUN397 dataset[35], [36].”
MethodAccurate, Dense, and Robust Multiview Stereopsis
Furukawa, Y., & Ponce, J. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.161 · Cited by 3,019
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568
Pixelwise View Selection for Unstructured Multi-View Stereo
Schönberger, J. L., Zheng, E., Frahm, J.-M., & Pollefeys, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_31 · Cited by 2,315
SUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · cited 1×
“Scene dataset, [2], [17],[1], and [18] do explore environments more than “table-top” data but do not have a dense set of views to simulate robot motion.”
BackgroundTorch7: A Matlab-like Environment for Machine Learning
Collobert, R., Kavukcuoglu, K., & Farabet, C. (2011) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,630
A large-scale hierarchical multi-view RGB-D object dataset
Lai, K., Bo, L., Ren, X., & Fox, D. (2011) · 2011 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2011.5980382 · Cited by 1,316
Bajcsy, R. (1988) · Proceedings of the IEEE · doi:10.1109/5.5968 · Cited by 830
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
Towards Internet-scale multi-view stereo
Furukawa, Y., Curless, B., Seitz, S. M., & Szeliski, R. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539802 · Cited by 751
Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
Su, H., Qi, C. R., Li, Y., & Guibas, L. J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.308 · Cited by 680
Unsupervised feature learning for 3D scene labeling
Lai, K., Bo, L., & Fox, D. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6907298 · Cited by 317 · cited 2×
“While other datasets do sample multiple images per scene, they often sample either from just a few positions or along only a few paths through the environment [1], [2].”
BackgroundBigBIRD: A large-scale 3D database of object instances
Singh, A., Sha, J., Narayan, K. S., Achim, T., & Abbeel, P. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6906903 · Cited by 311
Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd
Doumanoglou, A., Kouskouridas, R., Malassiotis, S., & Kim, T.-K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.390 · Cited by 242
Pairwise Decomposition of Image Sequences for Active Multi-view Recognition
Johns, E., Leutenegger, S., & Davison, A. J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.414 · Cited by 223
RGBD Datasets: Past, Present and Future
Firman, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.88 · Cited by 147
Fast Single Shot Detection and Pose Estimation
Poirson, P., Ammirato, P., Fu, C.-Y., Liu, W., Kosecka, J., & Berg, A. C. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.78 · Cited by 109
Self-citationLook-Ahead Before You Leap: End-to-End Active Recognition by Forecasting the Effect of Motion
Jayaraman, D., & Grauman, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_30 · Cited by 75 · cited 1×
“Our baseline action network is inspired by a recent active vision approach [37], [26].”
MethodMultiview RGB-D Dataset for Object Instance Detection
Georgakis, G., Reza, M. A., Mousavian, A., Le, P.-H., & Kosecka, J. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.52 · Cited by 70
Robust single-view instance recognition
Held, D., Thrun, S., & Savarese, S. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487365 · Cited by 41
Hypothesis testing framework for active object detection
Atanasov, N., Sankaran, B., Ny, J. L., Koletschka, T., Pappas, G. J., & Daniilidis, K. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6631173 · Cited by 33
Robot In a Room: Toward Perfect Object Recognition in Closed EnvironmentsPreprint
Song, S., Zhang, L., & Xiao, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.02703 · Cited by 32
Modelling Observation Correlations for Active Exploration and Robust Object Detection
Velez, J., Hemann, G., Huang, A. S., Posner, I., & Roy, N. (2012) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3516 · Cited by 18
Controlled Recognition Bounds for Visual Learning and Exploration
Karasev, V. I., Chiuso, A., & Soatto, S. (2012) · neural information processing systems · Cited by 17
Robotic object detection: learning to improve the classifiers using sparse graphs for path planning
Jia, Z., Saxena, A., & Chen, T. (2011) · doi:10.5591/978-1-57735-516-8/ijcai11-346 · Cited by 14
Modelling Observation Correlations for Active Exploration and Robust Object Detection
Velez, J. J., Huang, A. S., Hemann, G. A., Roy, N., & Roy, N. (2011) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 14
Research works printed but not linked: 1
- wiedemeyer, 0, IAI Kinect2, Institute for Artificial Intelligence University Bremen 2014–2015
Cited by
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237
“Datasets have been a key driver of progress in computer vision, NLP, and other areas of AI [10, 18, 4, 1].”
BackgroundAnderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
Gibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection
Dwibedi, D., Misra, I., & Hebert, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.146 · Cited by 651
Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision
Queralta, J. P., Taipalmaa, J., Can Pullinen, B., Sarker, V. K., Nguyen Gia, T., Tenhunen, H., Gabbouj, M., Raitoharju, J., & Westerlund, T. (2020) · IEEE Access · doi:10.1109/access.2020.3030190 · Cited by 526
Metadata
- Authors
- Ammirato, P., Poirson, P., Park, E., Kosecka, J., & Berg, A. C.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- ORCID
- Park, Eunbyung
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 1378-1385
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2593769898
- Cited by
- 187
- References
- 40
- Citations by year
- 2026: 3 · 2025: 24 · 2024: 22 · 2023: 12 · 2022: 21 · 2021: 28 · 2020: 27 · 2019: 26 · 2018: 21 · 2017: 3
- FWCI
- 6.16
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- active vision, robot vision, RGB-D, object detection benchmark, reinforcement learning, occlusion, viewing direction, indoor environment, object classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.