Deep multimodal embedding: Manipulating novel objects with point-clouds, language and trajectories
Open accessAbstract
A robot operating in a real-world environment needs to perform reasoning over a variety of sensor modalities such as vision, language and motion trajectories. However, it is extremely challenging to manually design features relating such disparate modalities. In this work, we introduce an algorithm that learns to embed point-cloud, natural language, and manipulation trajectory data into a shared embedding space with a deep neural network. To learn semantically meaningful spaces throughout our network, we use a loss-based margin to bring embeddings of relevant pairs closer together while driving less-relevant cases from different modalities further apart. We use this both to pre-train its lower layers and fine-tune our final embedding space, leading to a more robust representation. We test our algorithm on the task of manipulating novel objects and appliances based on prior experience with other objects. On a large dataset, we achieve significant improvements in both accuracy and inference time over the previous state of the art. We also perform end-to-end experiments on a PR2 robot utilizing our learned embedding space.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Neural networks are convenient and effective at combining multi-modal sensory inputs, as well as integrating passive and interactive perception.
“NNs are also convenient and effective at combining multi-modal sensory inputs [55], as well as integrating passive and interactive perception [56,57,58].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
ROS: an open-source Robot Operating System
Quigley, M. (2009) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · Cited by 8,729 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322 · see citing works
Distance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., & Saul, L. K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577078 · Cited by 4,651 · see citing works
(2011) · Cited by 2,752 · see citing works
Large Margin Methods for Structured and Interdependent Output Variables
Tsochantaridis, I., Joachims, T., Hofmann, T., & Altün, Y. (2005) · MPG.PuRe (Max Planck Society) · Cited by 1,879 · see citing works
Deep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2013) · Int. J. Robotics Res. · doi:10.1177/0278364914549607 · Cited by 1,820 · see citing works
Self-citationExploiting Similarities among Languages for Machine TranslationPreprint
Mikolov, T., Le, Q. V., & Sutskever, I. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1309.4168 · Cited by 1,502 · see citing works
Scalable Nearest Neighbor Algorithms for High Dimensional Data
Muja, M., & Lowe, D. G. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2321376 · Cited by 1,401 · see citing works
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
Socher, R., Pennington, J., Huang, E., Ng, A. Y., & Manning, C. D. (2011) · Cited by 1,181 · see citing works
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · doi:10.5555/2627435.2697059 · Cited by 1,070 · see citing works
Theano: new features and speed improvements
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., Warde-Farley, D., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5590 · Cited by 1,005 · see citing works
Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments
Rusu, R. B. (2010) · KI - Künstliche Intelligenz · doi:10.1007/s13218-010-0059-6 · Cited by 938 · see citing works
Discriminative Deep Metric Learning for Face Verification in the Wild
Hu, J., Lu, J., & Tan, Y. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.242 · Cited by 697 · see citing works
Wsabie: Scaling up to large vocabulary image annotation
Weston, J., Samy Bengio, A., & Usunier, N. (2011) · Cited by 659 · see citing works
On rectified linear units for speech processing
Zeiler, M., Ranzato, M., Monga, R., Mao, M. Z., Yang, K., Le, Q. V., Nguyen, P., Senior, A., Vanhoucke, V., Dean, J., et al. (2013) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2013.6638312 · Cited by 552 · see citing works
DeepMPC: Learning Deep Latent Features for Model Predictive Control
Lenz, I., Knepper, R. A., & Saxena, A. (2015) · Robotics: Science and Systems · doi:10.15607/rss.2015.xi.012 · Cited by 339 · see citing works
Self-citationMeasuring Invariances in Deep Networks
Goodfellow, I., Lee, H., Le, Q. V., Saxe, A., & Ng, A. Y. (2009) · Cited by 337 · see citing works
Leveraging big data for grasp planning
Kappler, D., Bohg, J., & Schaal, S. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7139793 · Cited by 284 · see citing works
Learning contact-rich manipulation skills with guided policy search
Levine, S., Wagener, N., & Abbeel, P. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7138994 · Cited by 276 · see citing works
A geometric approach to robotic laundry folding
Miller, S., van den Berg, J., Fritz, M., Darrell, T., Goldberg, K., & Abbeel, P. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911430417 · Cited by 238 · see citing works
Difference of Normals as a Multi-scale Operator in Unorganized Point Clouds
Ioannou, Y., Taati, B., Harrap, R., & Greenspan, M. (2012) · International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission · doi:10.1109/3dimpvt.2012.12 · Cited by 181 · see citing works
Improved Multimodal Deep Learning with Variation of Information
Sohn, K., Shang, W., & Lee, H. (2014) · neural information processing systems · Cited by 142 · see citing works
Deep Learning for Detecting Robotic Grasps
Lenz, I., Lee, H., & Saxena, A. (2013) · Robotics: Science and Systems IX · doi:10.15607/rss.2013.ix.012 · Cited by 126 · see citing works
Self-citationDeep belief net learning in a long-range vision system for autonomous off-road driving
Hadsell, R., Erkan, A., Sermanet, P., Scoffier, M., Muller, U., & LeCun, Y. (2008) · 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2008.4651217 · Cited by 101 · see citing works
A kernel-based approach to direct action perception
Kroemer, O., Ugur, E., Oztop, E., & Peters, J. (2012) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6224957 · Cited by 90 · see citing works
Pre- and post-contact policy decomposition for planar contact manipulation under uncertainty
Koval, M. C., Pollard, N. S., & Srinivasa, S. S. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364915594474 · Cited by 59 · see citing works
Tell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions
Kumar Misra, D., Sung, J., Lee, K., & Saxena, A. (2014) · Robotics: Science and Systems X · doi:10.15607/rss.2014.x.005 · Cited by 59 · see citing works
Self-citationData-Driven Online Decision Making for Autonomous Manipulation
Kappler, D., Pástor, P., Kalakrishnan, M., Wüthrich, M., & Stefan Schaal, A. (2015) · Robotics: Science and Systems · doi:10.15607/rss.2015.xi.044 · Cited by 57 · see citing works
Sung, J., Jin, S. H., & Saxena, A. (2018) · Springer Proceedings in Advanced Robotics · doi:10.1007/978-3-319-60916-4_40 · Cited by 52 · cited 10× · see citing works
“As an example application, we consider manipulating novel appliances [1].”
BackgroundMethodInfluentialSelf-citationBakebot: Baking Cookies with the PR2
Bollini, M. A., Barry, J. D., & Rus, D. L. (2011) · Cited by 48 · see citing works
Learning To Embed Songs And Tags For Playlist Prediction.
Moore, J. L., Chen, S., Joachims, T., & Turnbull, D. (2012) · doi:10.5281/zenodo.1416965 · Cited by 40 · see citing works
Synthesizing manipulation sequences for under-specified tasks using unrolled Markov Random Fields
Sung, J., Selman, B., & Saxena, A. (2013) · 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2014.6942972 · Cited by 22 · see citing works
Self-citationSynthesizing Manipulation Sequences for Under-Specified Tasks using Unrolled Markov Random Fields
Sung, J., Selman, B., & Saxena, A. (2014) · Cited by 13 · see citing works
Self-citationTransfer of object shape knowledge across visual and haptic modalities
Erdogan, G., Yildirim, I., & Jacobs, R. A. (2014) · eScholarship (California Digital Library) · doi:10.13140/2.1.1513.9206 · Cited by 6 · see citing works
Flagged first, then by citations
3 of the 41 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding
Wang, Z., Li, H., & Rajagopal, R. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5450 · Cited by 103 · see citing works
(2021) · Journal of Machine Learning Research · Cited by 224 · see citing works
Robotics Dexterous Grasping: The Methods Based on Point Cloud and Deep Learning
Duan, H., Wang, P., Huang, Y.-Y., Xu, G., Wei, W., & Shen, X. (2021) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2021.658280 · Cited by 61 · see citing works
Modeling Grasp Motor Imagery Through Deep Conditional Generative Models
Veres, M., Moussa, M. A., & Taylor, G. W. (2017) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2017.2651945 · Cited by 40 · see citing works
A deep learning gated architecture for UGV navigation robust to sensor failures
Patel, N., Choromanska, A., Krishnamurthy, P., & Khorrami, F. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.03.001 · Cited by 36 · see citing works
Safe and Efficient Robot Manipulation: Task-Oriented Environment Modeling and Object Pose Estimation
Zhang, Y., Tian, G., & Shao, X. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3071222 · Cited by 22 · see citing works
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“NNs are also convenient and effective at combining multi-modal sensory inputs [55], as well as integrating passive and interactive perception [56–58].”
BackgroundA survey of Semantic Reasoning frameworks for robotic systems
Liu, W., Daruna, A., Patel, M., Ramachandruni, K., & Chernova, S. (2022) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2022.104294 · Cited by 18 · see citing works
Deep metric learning for visual servoing: when pose and image meet in latent space
Felton, S., Fromont, É., & Marchand, É. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160963 · Cited by 10 · see citing works
Robinson, F., & Nejat, G. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161404 · Cited by 10 · see citing works
Park, S., & Kim, Y. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.10.009 · Cited by 10 · see citing works
Deep learning for multisensorial and multimodal interaction
Keren, G., Mousa, A. E.-D., Pietquin, O., Zafeiriou, S. P., & Schuller, B. W. (2018) · ACM eBooks · doi:10.1145/3107990.3107996 · Cited by 10 · see citing works
Policy Blending and Recombination for Multimodal Contact-Rich Tasks
Narita, T., & Kroemer, O. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3061982 · Cited by 9 · see citing works
Inferring 3D Shapes of Unknown Rigid Objects in Clutter Through Inverse Physics Reasoning
Song, C., & Boularias, A. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2885579 · Cited by 9 · see citing works
Ontology Generation Using GUI and Simulation for Service Robots to Operate Home Appliances
Nakamura, T., Yuguchi, A., Aubert, M., Ricardez, G. A. G., Takamatsu, J., & Ogasawara, T. (2019) · IEEE International Conference on Robotic Computing (IRC) · doi:10.1109/irc.2019.00058 · Cited by 7 · see citing works
Yuguchi, A., Nakamura, T., Toyoda, M., Yamada, M., Tulathum, P., Aubert, M., Ricardez, G. A. G., Takamatsu, J., & Ogasawara, T. (2022) · Advanced Robotics · doi:10.1080/01691864.2022.2070422 · Cited by 5 · see citing works
Shin, G., & Mun, S. (2023) · Developmental Science · doi:10.1111/desc.13405 · Cited by 5 · see citing works
Domain Adaptation from Public Dataset to Robotic Perception Based on Deep Neural Network
Yi, C., Chen, H., Hu, X., & Xu, Y. (2020) · Chinese Automation Congress · doi:10.1109/cac51589.2020.9327112 · Cited by 5 · see citing works
Multiscale Sensor Fusion and Continuous Control with Neural CDEs
Singh, S., Ramirez, F. M., Varley, J., Zeng, A., & Sindhwani, V. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9982210 · Cited by 3 · see citing works
One-Shot Imitation Learning on Heterogeneous Associated Tasks via Conjugate Task Graph
Huang, T., Zhao, F., & Wang, D. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533467 · Cited by 2 · see citing works
Grasping Novel Objects by Semi-supervised Domain Adaptation
Cai, J., Zhang, Z., & Cheng, H. (2019) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar47638.2019.9043954 · Cited by 1 · see citing works
Detecting Saliency by Combining Speech and Object Detection in Indoor Environments
Thapa, K. (2018) · doi:10.18122/td/1446/boisestate · Cited by 1 · see citing works
Kadi, H. A., & Terzić, K. (2022) · Preprints.org · doi:10.20944/preprints202212.0305.v1 · Cited by 1 · see citing works
Veres, M. (2016) · The Atrium (University of Guelph)
Reasoning on Grasp-Action Affordances
Ardón, P., Pairet, È., Petrick, R. P. A., Ramamoorthy, S., & Lohan, K. S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-23807-0_1
Adapting Movements and Behaviour to Favour Communication in Human-Robot Interaction
Lohan, K. S., Ahmad, M., Dondrup, C., Ardón, P., Pairet, È., & Vinciarelli, A. (2020) · Springer eBooks · doi:10.1007/978-3-030-46732-6_13
- Authors
- Sung, J., Lenz, I., & Saxena, A.
- Affiliations
- Cornell University (US)
- ORCID
- Saxena, Ashutosh
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 2794-2801
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2182599745
- Cited by
- 26
- References
- 41
- Citations by year
- 2023: 4 · 2022: 4 · 2021: 4 · 2020: 4 · 2019: 5 · 2018: 3 · 2017: 1 · 2016: 1
- FWCI
- 5.43
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- multimodal embeddings, point clouds, natural language, shared embedding space, margin-based loss, pre-training and fine-tuning, robotic manipulation, PR2 robot
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
