Leveraging appearance priors in non-rigid registration, with application to manipulation of deformable objects
Abstract
Manipulation of deformable objects is a widely applicable but challenging task in robotics. One promising nonparametric approach for this problem is trajectory transfer, in which a non-rigid registration is computed between the starting scene of the demonstration and the scene at test time. This registration is extrapolated to find a function from ℝ3to ℝ3, which is then used to warp the demonstrated robot trajectory to generate a proposed trajectory to execute in the test scene. In prior work [1] [2], only depth information from the scenes has been used to compute this warp function. This approach ignores appearance information, but there are situations in which using both shape and appearance information is necessary for finding high quality non-rigid warp functions. In this paper, we describe an approach to learn relevant appearance information about deformable objects using deep learning, and use this additional information to improve the quality of non-rigid registration between demonstration and test scenes. Our method better registers areas of interest on deformable objects that are crucial for manipulation, such as rope crossings and towel corners and edges. We experimentally validate our approach in both simulation and in the real world, and show that the utilization of appearance information leads to a significant improvement in both selecting the best matching demonstration scene for a given test scene, and finding a high quality non-rigid registration between those two scenes.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most knot tying systems act either on the table or in the air.
“Most KT systems act either on the table [243,244,275,282,286,287] or in the air [52,91,105,239,284,285] as shown in Figure 6.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [244]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · see citing works
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278 · see citing works
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Shape matching and object recognition using shape contexts
Belongie, S., Malik, J., & Puzicha, J. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.993558 · Cited by 6,306 · see citing works
Spline Models for Observational Data.
Müller, H., & Wahba, G. (1991) · Journal of the American Statistical Association · doi:10.2307/2290434 · Cited by 4,992 · see citing works
Spline Models for Observational Data
Wahba, G. (1990) · doi:10.1137/1.9781611970128 · Cited by 4,184 · see citing works
The PASCAL visual object classes challenge 2006 (VOC2006) results
Everingham, M., Zisserman, A., Williams, C. K. I., & Gool, L. V. (2006) · Cited by 2,079 · see citing works
Reconstruction and representation of 3D objects with radial basis functions
Carr, J. C., Beatson, R. K., Cherrie, J. B., Mitchell, T. J., Fright, W. R., McCallum, B. C., & Evans, T. R. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383266 · Cited by 1,852 · cited 1× · see citing works
“We set r(f) to be the Thin Plate Spline (TPS) regularizer [33] [34], a commonly-used effective regularizer that calculates the bending cost of f :”
MethodSIFT Flow: Dense Correspondence across Scenes and Its Applications
Liu, C., Yuen, J., & Torralba, A. B. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.147 · Cited by 1,590 · see citing works
A new point matching algorithm for non-rigid registration
Chui, H., & Rangarajan, A. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00009-2 · Cited by 1,560 · see citing works
DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo
Tola, E., Lepetit, V., & Fua, P. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.77 · Cited by 1,405 · see citing works
Robot Programming by Demonstration
Billard, A., Calinon, S., Dillmann, R., & Schaal, S. (2008) · Springer eBooks · doi:10.1007/978-3-540-30301-5_60 · Cited by 1,078 · see citing works
Joint Deep Learning for Pedestrian Detection
Ouyang, W., & Wang, X. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.257 · Cited by 665 · see citing works
Landmark-based elastic registration using approximating thin-plate splines
Rohr, K., Stiehl, H. S., Sprengel, R., Buzug, T. M., Weese, J., & Kuhn, M. H. (2001) · IEEE Transactions on Medical Imaging · doi:10.1109/42.929618 · Cited by 424 · see citing works
Deformable part models are convolutional neural networks
Girshick, R., Iandola, F., Darrell, T., & Malik, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298641 · Cited by 423 · see citing works
Maitin-Shepard, J., Cusumano-Towner, M., Lei, J., & Abbeel, P. (2010) · 2010 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2010.5509439 · Cited by 418 · see citing works
Self-citationRandomized Trees for Real-Time Keypoint Recognition
Lepetit, V., Lagger, P., & Fua, P. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.288 · Cited by 416 · see citing works
Histogram intersection kernel for image classification
Barla, A., Odone, F., & Verri, A. (2004) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2003.1247294 · Cited by 283 · see citing works
Reinforcement Learning in Robotics: Applications and Real-World Challenges
Kormushev, P. S., Calinon, S., & Caldwell, D. G. (2013) · Robotics · doi:10.3390/robotics2030122 · Cited by 259 · see citing works
van den Berg, J., Miller, S., Duckworth, D., Hu, H., Wan, A., Fu, X., Goldberg, K., & Abbeel, P. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509621 · Cited by 233 · see citing works
Self-citationAn obstacle-based rapidly-exploring random tree
Rodriguez, Tang, X., Lien, J., & Amato, N. M. (2006) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2006.1641823 · Cited by 207 · see citing works
Manipulation of deformable objects without modeling and simulating deformation
Berenson, D. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6697007 · Cited by 162 · see citing works
Needle Steering in 3-D Via Rapid Replanning
Patil, S., Burgner-Kahrs, J., Webster, R. J., III, & Alterovitz, R. (2014) · IEEE Transactions on Robotics · doi:10.1109/tro.2014.2307633 · Cited by 155 · see citing works
Learning from Demonstrations Through the Use of Non-rigid Registration
Schulman, J., Ho, J., Lee, C., & Abbeel, P. (2016) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-319-28872-7_20 · Cited by 133 · see citing works
Self-citationOn improving the extrapolation capability of task-parameterized movement models
Calinon, S., Alizadeh, T., & Caldwell, D. G. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6696414 · Cited by 102 · see citing works
Motion Planning for Robotic Manipulation of Deformable Linear Objects
Saha, M., Isto, P., & Latombe, J. (2008) · Springer eBooks · doi:10.1007/978-3-540-77457-0_3 · Cited by 101 · see citing works
Schulman, J., Gupta, A., Venkatesan, S., Tayson-Frederick, M., & Abbeel, P. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6696945 · Cited by 84 · see citing works
Self-citationMotion planning for robotic manipulation of deformable linear objects
Saha, M., & Isto, P. (2006) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2006.1642074 · Cited by 81 · see citing works
Dexterous Robotic Manipulation of Deformable Objects with Multi-Sensory Feedback - a Review
Fouad, F., & Payeur, P. (2010) · InTech eBooks · doi:10.5772/9183 · Cited by 69 · see citing works
Deformable Part Models are Convolutional Neural NetworksPreprint
Girshick, R., Iandola, F., Darrell, T., & Malik, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.5403 · Cited by 64 · see citing works
Combining imitation and reinforcement learning to fold deformable planar objects
Balaguer, B., & Carpin, S. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6094992 · Cited by 59 · see citing works
Deformation Planning for Robotic Soft Tissue Manipulation
Smolen, J., & Patriciu, A. (2009) · doi:10.1109/achi.2009.31 · Cited by 44 · see citing works
Efficient motion planning for manipulation robots in environments with deformable objects
Frank, B., Stachniss, C., Abdo, N., & Burgard, W. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6094946 · Cited by 43 · see citing works
Adaptive Control for Human-Robot Skilltransfer: Trajectory Planning Based on Fluid Dynamics
Mayer, H. G., Nagy, I., Knoll, A., Braun, E. U., Lange, R., & Bauernschmitt, R. (2007) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2007.363583 · Cited by 40 · see citing works
Landmark-Based Non-rigid Registration Via Graph Cuts
Lombaert, H., Sun, Y., & Chériet, F. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74260-9_15 · Cited by 16 · see citing works
A new efficient EM-ICP algorithm for non-linear registration of 3D point sets
Combès, B., & Prima, S. (2012) · Cited by 5 · see citing works
Flagged first, then by citations
3 of the 44 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Sánchez, J. M. O., Ramón, J. A. C., Bouzgarrou, B. C., & Mezouar, Y. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364918779698 · Cited by 454 · see citing works
Modeling, learning, perception, and control methods for deformable object manipulation
Yin, H., Varava, A., & Kragic, D. (2021) · Science Robotics · doi:10.1126/scirobotics.abd8803 · Cited by 299 · see citing works
Manipulating deformable objects by interleaving prediction, planning, and control
McConachie, D., Dobson, A., Ruan, M., & Berenson, D. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364920918299 · Cited by 61 · see citing works
Automated planning for robotic layup of composite prepreg
Malhan, R. K., Shembekar, A. V., Kabir, A. M., Bhatt, P. M., Shah, B. C., Zanio, S., Nutt, S., & Gupta, S. K. (2020) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2020.102020 · Cited by 58 · see citing works
Cloth Manipulation Using Random-Forest-Based Imitation Learning
Jia, B., Pan, Z., Hu, Z., Pan, J., & Manocha, D. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2897370 · Cited by 42 · see citing works
Self-citationBayesian nonparametric learning of cloth models for real-time state estimation
Koganti, N., Tamei, T., Ikeda, K., & Shibata, T. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2691721 · Cited by 41 · see citing works
Automated assembly of non-rigid objects
Makris, S., Dietrich, F., Kellens, K., & Hu, S. J. (2023) · CIRP Annals · doi:10.1016/j.cirp.2023.05.003 · Cited by 38 · see citing works
Deformable Linear Object Prediction Using Locally Linear Latent Dynamics
Zhang, W., Schmeckpeper, K., Chaudhari, P., & Daniilidis, K. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560955 · Cited by 29 · see citing works
Estimating Model Utility for Deformable Object Manipulation Using Multiarmed Bandit Methods
McConachie, D., & Berenson, D. (2018) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2018.2822669 · Cited by 29 · see citing works
Robotic manipulation of deformable objects by tangent space mapping and non-rigid registration
Tang, T., Liu, C., Chen, W., & Tomizuka, M. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759418 · Cited by 25 · see citing works
Bandit-Based Model Selection for Deformable Object Manipulation
McConachie, D., & Berenson, D. (2020) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-43089-4_45 · Cited by 25 · see citing works
Human-Guided Object Mapping for Task Transfer
Fitzgerald, T., Goel, A. K., & Thomaz, A. L. (2018) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3277905 · Cited by 21 · see citing works
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“Most KT systems act either on the table [243,244,275,282,286,287] or in the air [52,91,105,239,284,285] as shown in Figure 6.”
BackgroundMethodMotion planning for fluid manipulation using simplified dynamics
Pan, Z., & Manocha, D. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759622 · Cited by 18 · see citing works
An interactive simulator for deformable linear objects manipulation planning
Álvarez, N., & Yamazaki, K. (2016) · IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR) · doi:10.1109/simpar.2016.7862405 · Cited by 15 · see citing works
Interleaving Planning and Control for Deformable Object Manipulation
McConachie, D., Ruan, M., & Berenson, D. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_68 · Cited by 15 · see citing works
Haekal, M., Arimura, H., Hirose, T., Shibayama, Y., Ohga, S., Fukunaga, J., Umezu, Y., Honda, H., & Sasaki, T. (2018) · Physica Medica · doi:10.1016/j.ejmp.2017.12.019 · Cited by 16 · see citing works
Koganti, N., Shibata, T., Tamei, T., & Ikeda, K. (2019) · Advanced Robotics · doi:10.1080/01691864.2019.1610061 · Cited by 12 · see citing works
A novel cable-grasping planner for manipulator based on the operation surface
Wu, W. H., Zhu, Y., Zheng, X., & Guo, Y. H. (2021) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2021.102252 · Cited by 11 · see citing works
Guo, J., Zhang, J., Gai, Y., Wu, D., & Chen, K. (2022) · IEEE Information Technology and Mechatronics Engineering Conference (ITOEC) · doi:10.1109/itoec53115.2022.9734432 · Cited by 10 · see citing works
A Review of Sensors Used on Fabric-Handling Robots
Kaltsas, P. I., Koustoumpardis, P. Ν., & Nikolakopoulos, P. G. (2022) · Machines · doi:10.3390/machines10020101 · Cited by 9 · see citing works
A Simulation-Based Grasp Planner for Enabling Robotic Grasping during Composite Sheet Layup
Manyar, O. M., Desai, J., Deogaonkar, N., Joesph, R. J., Malhan, R. K., McNulty, Z., Wang, B., Barbič, J., & Gupta, S. K. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560939 · Cited by 8 · see citing works
Online Grasp Plan Refinement for Reducing Defects During Robotic Layup of Composite Prepreg Sheets
Malhan, R. K., Joseph, R. J., Shembekar, A. V., Kabir, A. M., Bhatt, P. M., & Gupta, S. K. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196876 · Cited by 8 · see citing works
Polymorphic robot learning for dynamic and contact-rich handling of soft-rigid objects
Lawhorn, R., Susanibar, S., Lu, L., & Wang, C. (2017) · IEEE International Conference on Advanced Intelligent Mechatronics (AIM) · doi:10.1109/aim.2017.8014082 · Cited by 7 · see citing works
Tracking linear deformable objects using slicing method
Rastegarpanah, A., Howard, R., & Stolkin, R. (2021) · Robotica · doi:10.1017/s0263574721001065 · Cited by 5 · see citing works
Tool path correction for robotic deburring using local non-rigid 3D registration
Peng, P., Wu, C., Yang, J., & Ding, H. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2024.102745 · Cited by 6 · see citing works
Bednarek, M., & Walas, K. (2019) · International Conference on Robotics and Automation Engineering (ICRAE) · doi:10.1109/icrae48301.2019.9043790 · Cited by 5 · see citing works
Tuli, S., Bansal, R., Paul, R., & Mausam, M. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13791 · Cited by 4 · see citing works
Synthesis of Robot Hand Skills Powered by Crowdsourced Learning
Zhao, L., Lawhorn, R., Wang, C., Lu, L., & Ouyang, B. (2019) · International Congress of Mathematicans · doi:10.1109/icmech.2019.8722953 · Cited by 3 · see citing works
Robot Composite Learning and the Nunchaku Flipping Challenge
Zhao, L., Zhao, Y., Patil, S., Davies, D., Wang, C., Lu, L., & Ouyang, B. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461141 · Cited by 3 · see citing works
Data-Oriented State Space Discretization for Crowdsourced Robot Learning of Physical Skills
Zhao, L., Lu, L., & Wang, C. (2020) · ASME Letters in Dynamic Systems and Control · doi:10.1115/1.4047961 · Cited by 2 · see citing works
Robot trajectory generation for three-dimensional flexible load transfer
Rahmouni, A., Lucet, E., Béarée, R., Olabi, A., & Grossard, M. (2019) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon.2019.8927150 · Cited by 2 · see citing works
Deformable Object Manipulation: Learning While Doing
McConachie, D. (2020) · Deep Blue (University of Michigan) · Cited by 2 · see citing works
Symplectic Integration for Multivariate Dynamic Spline-Based Model of Deformable Linear Objects
Khalifa, A., & Palli, G. (2021) · Journal of Computational and Nonlinear Dynamics · doi:10.1115/1.4052571 · Cited by 5 · see citing works
Deformable Linear Object Prediction Using Locally Linear Latent DynamicsPreprint
Zhang, W., Schmeckpeper, K., Chaudhari, P., & Daniilidis, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14184 · Cited by 1 · see citing works
Kadi, H. A., & Terzić, K. (2022) · Preprints.org · doi:10.20944/preprints202212.0305.v1 · Cited by 1 · see citing works
Shape Estimation of Viscoelastic Fluid Materials: An Application for Weld Pool
Assal, O. E., Mateo, C. M., Ciron, S., & David Fofi, B. (2024) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio64047.2024.10907566
- Authors
- Huang, S. H., Pan, J., Mulcaire, G., & Abbeel, P.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2015-09
- Pages
- 878-885
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W2206421543
- Cited by
- 39
- References
- 44
- Citations by year
- 2024: 2 · 2023: 2 · 2022: 5 · 2021: 8 · 2020: 6 · 2019: 6 · 2018: 5 · 2017: 2 · 2016: 3
- FWCI
- 4.67
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Keywords
- non-rigid registration, deformable object manipulation, deep learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
