CoChat
Sign Up
PreprintCited by 200Robot Manipulation and LearningReinforcement Learning in RoboticsAdvanced Neural Network Applications

Transporter Networks: Rearranging the Visual World for Robotic Manipulation

Open access
Zeng, Andy, Florence, Pete, Tompson, Jonathan, Welker, Stefan, Chien, Jonathan, Attarian, Maria, Armstrong, Travis, Krasin, Ivan +4 more·2020·arXiv
DOI 10.48550/arxiv.2010.14406arXiv 2010.14406
Open this paper in CoChat — free accountOpen PDF

Abstract

Robotic manipulation can be formulated as inducing a sequence of spatial displacements: where the space being moved can encompass an object, part of an object, or end effector. In this work, we propose the Transporter Network, a simple model architecture that rearranges deep features to infer spatial displacements from visual input - which can parameterize robot actions. It makes no assumptions of objectness (e.g. canonical poses, models, or keypoints), it exploits spatial symmetries, and is orders of magnitude more sample efficient than our benchmarked alternatives in learning vision-based manipulation tasks: from stacking a pyramid of blocks, to assembling kits with unseen objects; from manipulating deformable ropes, to pushing piles of small objects with closed-loop feedback. Our method can represent complex multi-modal policy distributions and generalizes to multi-step sequential tasks, as well as 6DoF pick-and-place. Experiments on 10 simulated tasks show that it learns faster and generalizes better than a variety of end-to-end baselines, including policies that use ground-truth object poses. We validate our methods with hardware in the real world. Experiment videos and code are available at https://transporternets.github.io

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.14406

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A large number of robot learning works and benchmarks use quasi-static tasks, making the system broadly applicable.

    “However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3-7,14,18,19,22,30,33,51, 55, 56, 64-66], making the system broadly applicable.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [19]

    Section 4.1 (page unavailable) describes Ravens as a benchmark of ten discrete-time tabletop manipulation tasks using position-controlled, two-pose motion primitives. This supports citing the paper as an example of a robot-learning benchmark with such tasks, but the copy does not call them quasi-static or establish how many works and benchmarks use quasi-static tasks. The sentence’s claim that this makes the citing authors’ system broadly applicable is their own inference. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

  • Some works have collected large datasets using pre-programmed demonstrators in simulation.

    “Some works have collected large datasets using pre-programmed demonstrators in simulation [18-22]”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [19]

  • Transporter Net solves vision-based pick-and-place behaviour cloning using neural networks.

    “Zeng et al. (2021) [104] propose Transporter Net to solve vision-based pick-and-place BC using NNs.”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [104]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. [1][1] S. Levine, C. Finn, T. Darrell, and P . Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research (JMLR) , 2016. -
  2. [2][2] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning (CoRL) , 2018. -
  3. [3][3] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. International Symposium on Robotics Research (ISRR) , 2019. -
  4. [4][4] T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V . Mnih. Unsupervised learning of object keypoints for perception and control. NeurIPS , 2019. -
  5. [5][5] A. Nagabandi, K. Konolige, S. Levine, and V . Kumar. Deep dynamics models for learning dexterous manipulation. Conference on Robot Learning (CoRL) , 2020. -
  6. [6][6] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige. Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020. -
  7. [7][7] O. Mees, N. Abdo, M. Mazuran, and W. Burgard. Metric learning for generalizing spatial relations to new objects. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2017. -
  8. [8][8] P. Jund, A. Eitel, N. Abdo, and W. Burgard. Optimization beyond the convolution: Generalizing spatial relations with end-to-end metric learning. IEEE International Conference on Robotics and Automation (ICRA) , 2018. -
  9. [9][9] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. Conference on Robot Learning (CoRL) , 2018. -
  10. [10][10] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters (RA-L) , 2019. -
  11. [11][11] P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. IEEE International Conference on Robotics and Automation (ICRA) , 2020. -
  12. [12][12] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. Workshop on Algorithmic Foundations of Robotics (WAFR) , 2020. -
  13. [13][13] R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. International Conference on Machine Learning (ICML) , 2018. -
  14. [14][14] T. Cohen and M. Welling. Group equivariant convolutional networks. International Conference on Machine Learning (ICML) , 2016. -
  15. [15][15] R. Platt, C. Kohler, and M. Gualtieri. Deictic image mapping: An abstraction for learning pose invariant manipulation policies. AAAI Conference on Artificial Intelligence , 2019. -
  16. [16][16] G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016. - · arXiv
  17. [17][17] Y. Yoon, G. N. DeSouza, and A. C. Kak. Real-time tracking and pose estimation for industrial objects using geometric features. IEEE International Conference on Robotics and Automation (ICRA) , 2003. -
  18. [18][18] M. Zhu, K. G. Derpanis, Y . Yang, S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis. Single image 3d object detection and pose estimation for grasping. ICRA , 2014. -
  19. [19][19] A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. IEEE International Conference on Robotics and Automation (ICRA) , 2017. -
  20. [20][20] X. Deng, Y . Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. IEEE International Conference on Robotics and Automation (ICRA) , 2020. -
  21. [21][21] H. Wang, S. Sridhar, J. Huang, J. V alentin, S. Song, and L. J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. Computer Vision and Pattern Recognition (CVPR) , 2019. -
  22. [22][22] E. Frazzoli, M. A. Dahleh, and E. Feron. Maneuver-based motion planning for nonlinear systems with symmetries. IEEE Transactions on Robotics (T-RO) , 2005. -
  23. [23][23] M. Gualtieri, A. ten Pas, and R. Platt. Pick and place without geometric object models. IEEE International Conference on Robotics and Automation (ICRA) , 2018. -
  24. [24][24] K. Zakka, A. Zeng, J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. IEEE International Conference on Robotics and Automation (ICRA) , 2020. -
  25. [25][25] A. Hundt, B. Killeen, N. Greene, H. Wu, H. Kwon, C. Paxton, and G. D. Hager. 'good robot!': Efficient reinforcement learning for multi-step visual tasks with sim to real transfer. IEEE Robotics and Automation Letters (RA-L) , 2020. -
  26. [26][26] L. Berscheid, P . Meißner, and T. Kr¨ oger. Self-supervised learning for precise pick-and-place without object model. IEEE Robotics and Automation Letters (RA-L) , 2020. -
  27. [28][28] C. Devin, P . Rowghanian, C. Vigorito, W. Richards, and K. Rohanimanesh. Self-supervised goal-conditioned pick and place. arXiv preprint arXiv:2008.11466 , 2020. - · arXiv
  28. [29][29] M. Khansari, D. Kappler, J. Luo, J. Bingham, and M. Kalakrishnan. Action image representation: Learning scalable deep grasping policies with zero real world data. IEEE International Conference on Robotics and Automation (ICRA) , 2020. -
  29. [30][30] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters (RA-L) , 2020. -
  30. [31][31] A. Zeng, S. Song, K.-T. Y u, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. International Conference on Robotics and Automation (ICRA) , 2018. -
  31. [34][34] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015. -
  32. [35][35] A. Zeng. Learning Visual Affordances for Robotic Manipulation . PhD thesis, Princeton University. -
  33. [36][36] B. Li, W. Wu, Q. Wang, F. Zhang, J. Xing, and J. Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. Conference on Computer Vision and Pattern Recognition , 2019. -
  34. [38][38] R. Tao, E. Gavves, and A. W. Smeulders. Siamese instance search for tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. -
  35. [40][40] D. Seita, P. Florence, J. Tompson, E. Coumans, V . Sindhwani, K. Goldberg, and A. Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. arXiv preprint arXiv:2012.03385 , 2020. - · arXiv
  36. [42][42] F. Yu and V . Koltun. Multi-scale context aggregation by dilated convolutions. International Conference on Learning Representations , 2016. -
  37. [43][43] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. International Conference on Machine Learning (ICML) , 2010. -
  38. [47][47] T. Schmidt, R. Newcombe, and D. Fox. Self-supervised visual descriptor learning for dense correspondence. IEEE Robotics and Automation Letters (RA-L) , 2016. -
  39. [48][48] R. Goroshin, M. F. Mathieu, and Y . LeCun. Learning to linearize under uncertainty. Advances in Neural Information Processing Systems (NeurIPS) , 2015. -
  40. [49][49] C. Szegedy, W. Liu, Y . Jia, P . Sermanet, S. Reed, D. Anguelov, D. Erhan, V . V anhoucke, and A. Rabinovich. Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015. -
  41. [50][50] R. Rahmatizadeh, P . Abolghasemi, L. B¨ ol¨ oni, and S. Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. ICRA , 2018. -
  42. [51][51] B. Mildenhall, P . P . Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European Conference on Computer Vision (ECCV) , 2020. -
  43. [52][52] V. Sitzmann, J. N. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems (NeurIPS) , 2020. -
  44. [53][53] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR) , 2015.

Cited by

Sorted by citations

Authors
Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong, T., Krasin, I., Duong, D., Wahid, A., Sindhwani, V., & Lee, J.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2010.14406
arXiv
2010.14406
OpenAlex
W3096099141
Cited by
200
References
4
Citations by year
2026: 1 · 2025: 22 · 2024: 55 · 2023: 57 · 2022: 42 · 2021: 22 · 2020: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
robotic manipulation, vision-based manipulation, spatial symmetry, pick-and-place, sample efficiency, closed-loop feedback

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review