The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Multi-view self-supervised deep learning for 6D pose estimation in the Amazon Picking Challenge
Open accessAbstract
Robot warehouse automation has attracted significant interest in recent years, perhaps most visibly in the Amazon Picking Challenge (APC) [1]. A fully autonomous warehouse pick-and-place system requires robust vision that reliably recognizes and locates objects amid cluttered environments, self-occlusions, sensor noise, and a large variety of objects. In this paper we present an approach that leverages multiview RGB-D data and self-supervised, data-driven learning to overcome those difficulties. The approach was part of the MIT-Princeton Team system that took 3rd- and 4th-place in the stowing and picking tasks, respectively at APC 2016. In the proposed approach, we segment and label multiple views of a scene with a fully convolutional neural network, and then fit pre-scanned 3D object models to the resulting segmentation to get the 6D object pose. Training a deep neural network for segmentation typically requires a large amount of training data. We propose a self-supervised method to generate a large labeled dataset without tedious manual segmentation. We demonstrate that our system can reliably estimate the 6D pose of objects under a variety of scenarios. All code, data, and benchmarks are available at http://apc.cs.princeton.edu/
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
A method for registration of 3-D shapes
Besl, P. J., & McKay, N. D. (1992) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.121791 · Cited by 18,103
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528
Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes
Hinterstoisser, S., Holzer, S., Cagniart, C., Ilic, S., Konolige, K., Navab, N., & Lepetit, V. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126326 · Cited by 643
Parametric estimate of intensity inhomogeneities applied to MRI
Styner, M., Brechbuhler, C., Szckely, G., & Gerig, G. (2000) · IEEE Transactions on Medical Imaging · doi:10.1109/42.845174 · Cited by 527
Analysis and Observations From the First Amazon Picking Challenge
Correll, N., Bekris, K. E., Berenson, D., Brock, O., Causo, A., Hauser, K., Okada, K., Rodriguez, A., Romano, J. M., & Wurman, P. R. (2018) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2016.2600527 · Cited by 425 · cited 1×
“Existing frameworks such as LINEMOD [12] or MOPED [13] work well under certain assumptions such as objects sitting on a table top with good illumination, but underperform when confronted with the limited visibility, shadows, and clutter imposed by the APC scenario [14].”
BackgroundThe MOPED framework: Object recognition and pose estimation for manipulation
Collet, A., Martinez, M., & Srinivasa, S. S. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911401765 · Cited by 421
Cited by
Deep Learning for 3D Point Clouds: A Survey
Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L., & Bennamoun, M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3005434 · Cited by 2,398
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Xiang, Y., Schmidt, T., Narayanan, V., & Fox, D. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.019 · Cited by 2,139
A survey on deep learning techniques for image and video semantic segmentation
Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., & Garcia-Rodriguez, J. (2018) · Applied Soft Computing · doi:10.1016/j.asoc.2018.05.018 · Cited by 1,490
Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Aparicio, J., & Goldberg, K. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.058 · Cited by 1,268
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
Zeng, A., Song, S., Niessner, M., Fisher, M., Xiao, J., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.29 · Cited by 1,128
Self-citationA Review on Deep Learning Techniques Applied to Semantic Segmentation
García-García, A., Orts‐Escolano, S., Oprea, S., Villena-Martínez, V., & García‐Rodríguez, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06857 · Cited by 991
TEASER: Fast and Certifiable Point Cloud Registration
Yang, H., Shi, J., & Carlone, L. (2021) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.3033695 · Cited by 895
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation
Wang, H., Sridhar, S., Huang, J., Valentin, J., Song, S., & Guibas, L. J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00275 · Cited by 757
Self-citationCut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection
Dwibedi, D., Misra, I., & Hebert, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.146 · Cited by 651
Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593986 · Cited by 606
Self-citationDeepIM: Deep Iterative Matching for 6D Pose Estimation
Li, Y., Wang, G., Ji, X., Xiang, Y., & Fox, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_42 · Cited by 576
Du, G., Wang, K., Lian, S., & Zhao, K. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-020-09888-5 · Cited by 475
GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation
Wang, G., Manhardt, F., Tombari, F., & Ji, X. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01634 · Cited by 462
Zeng, A., Song, S., Yu, K.-T., Donlon, E., Hogan, F. R., Bauza, M., Ma, D., Taylor, O., Liu, M., Romo, E., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461044 · Cited by 441
Self-citationPointNetGPD: Detecting Grasp Configurations from Point Sets
Liang, H., Ma, X., Li, S., Gorner, M., Tang, S., Fang, B., Sun, F., & Zhang, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794435 · Cited by 353
GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger
Donlon, E., Dong, S., Liu, M., Li, J., Adelson, E., & Rodriguez, A. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593661 · Cited by 332
Self-citationA Survey on Learning-Based Robotic Grasping
Kleeberger, K., Bormann, R., Kraus, W., & Huber, M. F. (2020) · Current Robotics Reports · doi:10.1007/s43154-020-00021-6 · Cited by 290
Robot Learning from Demonstration in Robotic Assembly: A Survey
Zhu, Z., & Hu, H. (2018) · Robotics · doi:10.3390/robotics7020017 · Cited by 255
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223
Transporter Networks: Rearranging the Visual World for Robotic Manipulation
Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong, T., Krasin, I., Duong, D., Wahid, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14406 · Cited by 200
Self-citationGrasping in the Wild: Learning 6DoF Closed-Loop Grasping From Low-Cost Demonstrations
Song, S., Zeng, A., Lee, J., & Funkhouser, T. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.3004787 · Cited by 185
“Classic vision-based grasping solutions often explicitly model contact forces with prior knowledge of object geometry, pose, and dynamics [8]–[11].”
BackgroundSelf-citationDeep learning in robotics: survey on model structures and training strategies
Karoly, A. I., Galambos, P., Kuti, J., & Rudas, I. J. (2021) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3018325 · Cited by 146
A novel weakly-supervised approach for RGB-D-based nuclear waste object detection
Sun, L., Zhao, C., Yan, Z., Liu, P., Duckett, T., & Stolkin, R. (2019) · IEEE Sensors Journal · doi:10.1109/jsen.2018.2888815 · Cited by 119
“In contrast, weakly-supervised deep learning has, so far, only achieved success in very few applications, including path planing [24] and the Amazon picking challenge [25].”
BackgroundD’Avella, S., Tripicchio, P., & Avizzano, C. A. (2020) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2019.101888 · Cited by 79
Fujita, M., Domae, Y., Noda, A., Garcia Ricardez, G. A., Nagatani, T., Zeng, A., Song, S., Rodriguez, A., Causo, A., Chen, I. M., et al. (2019) · Advanced Robotics · doi:10.1080/01691864.2019.1698463 · Cited by 78
“If a 3D model is not available, a method by the Team MIT-Princeton [35], which assumes the object pose by fitting primitive shapes like spheres and cubes directly to the data, is also proposed.”
MethodSelf-citationMulti-view 6D Object Pose Estimation and Camera Motion Planning Using RGBD Images
Sock, J., Kasaei, S. H., Lopes, L. S., & Kim, T.-K. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.260 · Cited by 60
Metadata
- Authors
- Zeng, A., Yu, K.-T., Song, S., Suo, D., Walker, E., Rodriguez, A., & Xiao, J.
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 1386-1383
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2963678509
- Cited by
- 457
- References
- 19
- FWCI
- 91.60
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.