Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations
Open accessAbstract
Grasp detection in clutter requires the robot to reason about the 3D scene from incomplete and noisy perception.In this work, we draw insight that 3D reconstruction and grasp learning are two intimately connected tasks, both of which require a fine-grained understanding of local geometry details.We thus propose to utilize the synergies between grasp affordance and 3D reconstruction through multi-task learning of a shared representation.Our model takes advantage of deep implicit functions, a continuous and memory-efficient representation, to enable differentiable training of both tasks.We train the model on self-supervised grasp trials data in simulation.Evaluation is conducted on a clutter removal task, where the robot clears cluttered objects by grasping them one at a time.The experimental results in simulation and on the real robot have demonstrated that the use of implicit neural representations and joint learning of grasp affordance and 3D reconstruction have led to state-ofthe-art grasping results.Our method outperforms baselines by over 10% in terms of grasp success rate.Additional results and videos can be found at https://sites.google.com/view/rpl-giga2021
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Marching cubes: A high resolution 3D surface construction algorithm
Lorensen, W. E., & Cline, H. E. (1987) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/37401.37422 · Cited by 10,143 · see citing works
Marching cubes: A high resolution 3D surface construction algorithm
Lorensen, W. E., & Cline, H. E. (1987) · ACM SIGGRAPH Computer Graphics · doi:10.1145/37402.37422 · Cited by 8,726 · see citing works
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_24 · Cited by 5,351 · see citing works
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
Park, J. J., Florence, P., Straub, J., Newcombe, R., & Lovegrove, S. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00025 · Cited by 3,557 · see citing works
A volumetric method for building complex models from range images
Curless, B., & Levoy, M. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237269 · Cited by 3,124 · see citing works
Occupancy Networks: Learning 3D Reconstruction in Function Space
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00459 · Cited by 2,733 · see citing works
Learning Implicit Fields for Generative Shape Modeling
Chen, Z., & Zhang, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00609 · Cited by 1,477 · see citing works
Implicit Neural Representations with Periodic Activation FunctionsPreprint
Sitzmann, V., Martel, J., Bergman, A. W., Lindell, D. B., & Wetzstein, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09661 · Cited by 1,456 · see citing works
Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Aparicio, J., & Goldberg, K. Y. (2017) · Robotics: Science and Systems · doi:10.15607/rss.2017.xiii.058 · Cited by 1,299 · see citing works
Open3D: A Modern Library for 3D Data ProcessingPreprint
Zhou, Q.-Y., Park, J., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09847 · Cited by 1,126 · see citing works
Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487517 · Cited by 1,060 · see citing works
Ferrari, C., & Canny, J. (2003) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1992.219918 · Cited by 916 · see citing works
Convolutional Occupancy Networks
Peng, S., Niemeyer, M., Mescheder, L., Pollefeys, M., & Geiger, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_31 · Cited by 845 · see citing works
The YCB object and model set: Towards common benchmarks for manipulation research
Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., & Dollar, A. M. (2015) · 2015 International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar.2015.7251504 · Cited by 838 · see citing works
Neural Sparse Voxel FieldsPreprint
Liu, L., Gu, J., Lin, K. Z., Chua, T., & Theobalt, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.11571 · Cited by 708 · see citing works
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697 · see citing works
Efficient grasping from RGBD images: Learning using a new rectangle representation
Yun, J., Moseson, S., & Saxena, A. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5980145 · Cited by 677 · see citing works
Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach
Morrison, D., Leitner, J., & Corke, P. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.021 · Cited by 662 · see citing works
Grasp Pose Detection in Point Clouds
ten Pas, A., Gualtieri, M., Saenko, K., & Platt, R. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364917735594 · Cited by 610 · see citing works
NeRF: Representing Scenes as Neural Radiance Fields for View SynthesisPreprint
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.08934 · Cited by 610 · see citing works
6-DOF GraspNet: Variational Grasp Generation for Object Manipulation
Mousavian, A., Eppner, C., & Fox, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00299 · Cited by 585 · see citing works
Learning ambidextrous robot grasping policies
Mahler, J., Matl, M., Satish, V., Danielczuk, M., DeRose, B., McKinley, S., & Goldberg, K. (2019) · Science Robotics · doi:10.1126/scirobotics.aau4984 · Cited by 567 · see citing works
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion
Chibane, J., Alldieck, T., & Pons‐Moll, G. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00700 · Cited by 534 · see citing works
1Data-Driven Grasp Synthesis- A SurveyPreprint
Bohg, J., Morales, A., Asfour, T., & Kragić, D. (2014) · Cited by 532 · see citing works
An overview of 3D object grasp synthesis algorithms
Sahbani, A., El-Khoury, S., & Bidaud, P. (2011) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2011.07.016 · Cited by 479 · see citing works
PointNetGPD: Detecting Grasp Configurations from Point Sets
Liang, H., Ma, X., Li, S., Gorner, M., Tang, S., Fang, B., Sun, F., & Zhang, J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794435 · Cited by 416 · see citing works
Local Deep Implicit Functions for 3D Shape
Genova, K., Cole, F. H., Sud, A., Sarna, A., & Funkhouser, T. A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00491 · Cited by 370 · see citing works
BigBIRD: A large-scale 3D database of object instances
Singh, A., Sha, J., Narayan, K. S., Achim, T., & Abbeel, P. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6906903 · Cited by 313 · see citing works
High precision grasp pose detection in dense clutter
Gualtieri, M., ten Pas, A., Saenko, K., & Platt, R. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759114 · Cited by 309 · see citing works
Leveraging big data for grasp planning
Kappler, D., Bohg, J., & Schaal, S. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7139793 · Cited by 284 · see citing works
Mahler, J., Liang, J., Niyaz, S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09312 · Cited by 277 · see citing works
Learning task-oriented grasping for tool manipulation from simulated self-supervision
Fang, K., Zhu, Y., Garg, A., Kurenkov, A., Mehta, V., Fei-Fei, L., & Savarese, S. (2018) · Robotics: Science and Systems Conference · doi:10.1177/0278364919872545 · Cited by 262 · see citing works
Self-citationLiu, Y. (1999) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.744611 · Cited by 256 · see citing works
DISN: Deep Implicit Surface Network for High-quality Single-view 3D ReconstructionPreprint
Xu, Q., Wang, W., Ceylan, D., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10711 · Cited by 236 · see citing works
Easily computable optimum grasps in 2-D and 3-D
Mirtich, B., & Canny, J. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1994.351399 · Cited by 235 · see citing works
Kasper, A., Xue, Z., & Dillmann, R. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364912445831 · Cited by 235 · see citing works
An Efficient RANSAC for 3D Object Recognition in Noisy and Occluded Scenes
Papazov, C., & Burschka, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-19315-6_11 · Cited by 166 · see citing works
Learning grasping points with shape context
Bohg, J., & Kragić, D. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2009.10.003 · Cited by 139 · see citing works
Learning 6-DOF Grasping Interaction via Deep Geometry-Aware 3D Representations
Yan, X., Hsu, J., Khansari, M., et al. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460609 · Cited by 114 · see citing works
Real-time tracking meets online grasp planning
Kragić, D., Miller, A. T., & Allen, P. K. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2001.932992 · Cited by 111 · see citing works
Affordance detection for task-specific grasping using deep learning
Kokic, M., Stork, J. A., Haustein, J. A., et al. (2017) · IEEE-RAS International Conference on Humanoid Robots · doi:10.1109/humanoids.2017.8239542 · Cited by 109 · see citing works
Mind the gap - robotic grasping under incomplete observation
Bohg, J., Johnson‐Roberson, M., León, B., et al. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5980354 · Cited by 106 · see citing works
On characterizing and computing three- and four-finger force-closure grasps of polyhedral objects
Ponce, J., Sullivan, S., Boissonnat, J. D., et al. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1993.291933 · Cited by 95 · see citing works
Learning Continuous 3D Reconstructions for Geometrically Aware Grasping
der Merwe, M. V., Lu, Q., Sundaralingam, B., Matak, M., & Hermans, T. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196981 · Cited by 76 · see citing works
Real-time CAD model matching for mobile manipulation and grasping
Klank, U., Pangercic, D., Rusu, R. B., et al. (2009) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2009.5379561 · Cited by 74 · see citing works
Learning to See before Learning to Act: Visual Pre-training for Manipulation
Lin, Y.-C., Zeng, A., Song, S., Isola, P., & Lin, T.-Y. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197331 · Cited by 66 · see citing works
Learning grasps for unknown objects in cluttered scenes
Fischinger, D., Vincze, M., & Jiang, Y. P. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630636 · Cited by 65 · see citing works
Learning Implicit Fields for Generative Shape ModelingPreprint
Chen, Z., & Zhang, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02822 · Cited by 59 · see citing works
Planning force-closure grasps on 3-D objects
Zhu, X., & Ding, H. (2004) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2004.1307997 · Cited by 51 · see citing works
Computing 3-D optimal form-closure grasps
Ding, D., Liu, Y.-H., & Wang, S. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2000.845288 · Cited by 47 · see citing works
Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in ClutterPreprint
Breyer, M., Chung, J. J., Ott, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.01132 · Cited by 42 · see citing works
Exploiting symmetries and extrusions for grasping household objects
Quispe, A. H., Milville, B., Gutierrez, M. A., et al. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7139713 · Cited by 35 · see citing works
Affordance-Based Grasping and Manipulation in Real World Applications
Pohl, C., Hitzler, K., Grimm, R., Zea, A., Hanebeck, U. D., & Asfour, T. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341482 · Cited by 27 · see citing works
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-SupervisionPreprint
Fang, K., Zhu, Y., Garg, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09266 · Cited by 13 · see citing works
Self-citationConvolutional Occupancy NetworksPreprint
Peng, S., Niemeyer, M., Mescheder, L., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04618 · Cited by 11 · see citing works
Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D RepresentationsPreprint
Yan, X., Hsu, J., Khansari, M., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.07303 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim- to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 12627-12637, 2019.
- Alberto Rodriguez, Matthew T Mason, and Steve Ferry. From caging to grasping. The International Journal of Robotics Research, 31(7), 2012.
Cited by
Sorted by citations
Ditto: Building Digital Twins of Articulated Objects from Interaction
Jiang, Z., Hsu, C.-C., & Zhu, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00553 · Cited by 92 · see citing works
Self-citationDitto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception
Hsu, C.-C., Jiang, Z., & Zhu, Y. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161431 · Cited by 21 · see citing works
Self-citationRoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., & Fox, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10721 · Cited by 7 · see citing works
- Authors
- Jiang, Z., Zhu, Y., Svetlik, M., Fang, K., & Zhu, Y.
- Affiliations
- The University of Texas at Austin (US)
- Venue
- Robotics: Science and Systems XVII
- Published
- 2021-07-12
- OpenAlex
- W3175388581
- Cited by
- 122
- References
- 60
- Citations by year
- 2026: 10 · 2025: 27 · 2024: 31 · 2023: 33 · 2022: 19 · 2021: 2
- FWCI
- 31.71
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Robotics: Science and Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1955523)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- grasp detection, implicit neural representations, 3D reconstruction, grasp affordances, multi-task learning, shared representations, clutter suppression, robotic manipulation, grasp success rate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
