CoChat
Sign Up
PreprintCited by 697Robot Manipulation and LearningReinforcement Learning in RoboticsSoft Robotics and Applications

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

Open access
Kalashnikov, Dmitry, Irpan, Alex, Pastor, Peter, Ibarz, Julian, Herzog, Alexander, Jang, Eric, Quillen, Deirdre, Holly, Ethan +3 more·2018·arXiv
DOI 10.48550/arxiv.1806.10293arXiv 1806.10293
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of grasping, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, whereby the robot continuously updates its grasp strategy based on the most recent observations to optimize long-horizon grasp success. To that end, we introduce QT-Opt, a scalable self-supervised vision-based reinforcement learning framework that can leverage over 580k real-world grasp attempts to train a deep neural network Q-function with over 1.2M parameters to perform closed-loop, real-world grasping that generalizes to 96% grasp success on unseen objects. Aside from attaining a very high success rate, our method exhibits behaviors that are quite distinct from more standard grasping systems: using only RGB vision-based perception from an over-the-shoulder camera, our method automatically learns regrasping strategies, probes objects to find the most effective grasps, learns to reposition objects and perform other non-prehensile pre-grasp manipulations, and responds dynamically to disturbances and perturbations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.10293

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A large number of robot learning works and benchmarks use quasi-static tasks, making the system broadly applicable.

    “However, a large number of robot learning works and benchmarks use quasi-static tasks [1,3-7,14,18,19,22,30,33,51, 55, 56, 64-66], making the system broadly applicable.”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [14]

    Related Work (p. 3) says current grasping systems typically predict a grasp pose and execute an open-loop plan, supporting the point that static approaches are common. But the paper does not establish that a large number of robot-learning benchmarks use quasi-static tasks; it presents QT-Opt as dynamic, closed-loop grasping that responds to disturbances (Discussion, p. 8). (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 38

  1. [1][1] J. Peters and S. Schaal. Reinforcement learning of motor skills with policy gradients. Neural Networks , 21(4):682 - 697, 2008. ISSN 0893-6080. Robotics and Neuroscience. -
  2. [2][2] M. Kalakrishnan, L. Righetti, P. Pastor, and S. Schaal. Learning Force Control Policies for Compliant Manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems , 2011. -
  3. [3][3] A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In IEEE/RSJ International Conference on Intelligent Robots and Systems , 2017. -
  4. [4][4] A. Ghadirzadeh, A. Maki, D. Kragic, and M. Bjrkman. Deep predictive policy training using reinforcement learning. In IEEE/RSJ International Conference on Intelligent Robots and Systems , 2017. -
  5. [6][6] D. Morrison et. al. Cartman: The low-cost Cartesian Manipulator that won the Amazon Robotics Challenge. In IEEE International Conference on Robotics and Automation , 2018. -
  6. [7][7] J. Mahler, M. Matl, X. Liu, A. Li, D. V. Gealy, and K. Goldberg. Dex-Net 3.0: Computing Robust Robot Suction Grasp Targets in Point Clouds using a New Analytic Model and Deep Learning. CoRR , abs/1709.06670, 2017. URL http://arxiv.org/abs/1709.06670 . - · arXiv
  7. [8][8] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp Pose Detection in Point Clouds. The International Journal of Robotics Research , 36(13-14):1455-1473, 2017. -
  8. [9][9] N. Chavan-Dafle and A. Rodriguez. Stable Prehensile Pushing: In-Hand Manipulation with Alternating Sticking Contacts. In IEEE Intl Conference on Robotics and Automation , 2018. -
  9. [10][10] J. Bohg, A. Morales, T. Asfour, and D. Kragic. Data-Driven Grasp Synthesis ASurvey. IEEE Transactions on Robotics , 30(2):289-309, 2014. -
  10. [11][11] R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning . MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981. -
  11. [15][15] R. Hafner and M. Riedmiller. Reinforcement learning in feedback control. Machine Learning , 84(1-2), 2011. -
  12. [16][16] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous Control with Deep Reinforcement Learning. CoRR , abs/1509.02971, 2015. URL http://arxiv.org/abs/1509.02971 . - · arXiv
  13. [17][17] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking Deep Reinforcement Learning for Continuous Control. In Intl Conference on Machine Learning , 2016. -
  14. [19][19] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation. In IEEE International Conference on Robotics and Automation , 2018. -
  15. [20][20] D. Quillen, E. Jang, O. Nachum, C. Finn, J. Ibarz, and S. Levine. Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods. In IEEE International Conference on Robotics and Automation , 2018. -
  16. [21][21] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end Training of Deep Visuomotor Policies. Journal of Machine Learning Research , 17(39), 2016. -
  17. [22][22] J. Weisz and P. K. Allen. Pose Error Robust Grasping from Contact Wrench Space Metrics. In IEEE International Conference on Robotics and Automation , 2012. -
  18. [23][23] I. Lenz, H. Lee, and A. Saxena. Deep Learning for Detecting Robotic Grasps. The International Journal of Robotics Research , 34(4-5):705-724, 2015. -
  19. [24][24] K. Yu and A. Rodriguez. Realtime State Estimation with Tactile and Visual sensing. Application to Planar Manipulation. In IEEE Intl Conference on Robotics and Automation , 2018. -
  20. [25][25] U. Viereck, A. ten Pas, K. Saenko, and R. Platt. Learning a visuomotor controller for real world robotic grasping using simulated depth images. In CoRL , 2017. -
  21. [26][26] K. Hausman, Y. Chebotar, O. Kroemer, G. S. Sukhatme, and S. Schaal. Regrasping using Tactile Perception and Supervised Policy Learning. In AAAI Symposium on Interactive MultiSensory Object Perception for Embodied Agents , 2017. -
  22. [27][27] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with large-scale data collection. In International Symposium on Experimental Robotics , 2016. -
  23. [28][28] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours. In IEEE International Conference on Robotics and Automation , 2016. -
  24. [30][30] V. Mnih et al. Human-level control through deep reinforcement learning. Nature , 518(7540): 529-533, 2015. -
  25. [31][31] S. Gu, T. Lillicrap, I. Sutskever, and S. Levine. Continuous Deep Q-learning with Model-based Acceleration. In Proceedings of Intl Conference on Machine Learning , 2016. -
  26. [32][32] B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838-855, 1992. -
  27. [33][33] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations , 2016. -
  28. [34][34] H. V. Hasselt. Double Q-learning. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems , 2010. -
  29. [36][36] S. Fujimoto, H. van Hoof, and D. Meger. Addressing Function Approximation Error in ActorCritic Methods. CoRR , 2018. URL http://arxiv.org/abs/1802.09477 . - · arXiv
  30. [37][37] B. Amos, L. Xu, and J. Z. Kolter. Input convex neural networks. In International Conference on Machine Learning , volume 70, pages 146-155, 2017. -
  31. [38][38] R. Rubinstein and D. Kroese. The Cross-Entropy Method . Springer-Verlag, 2004. -
  32. [40][40] A. X. Chang, T. A. Funkhouser, L. J. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rich 3d model repository. CoRR , abs/1512.03012, 2015. URL http://arxiv.org/abs/1512.03012 . - · arXiv
  33. [41][41] S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning , 2015. -
  34. [42][42] A. Stooke and P. Abbeel. Accelerated Methods for Deep Reinforcement Learning. CoRR , abs/1803.02811, 2018. URL http://arxiv.org/abs/1803.02811 . - · arXiv
  35. [43][43] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: scalable distributed deeprl with importance weighted actor-learner architectures. CoRR , abs/1802.01561, 2018. URL http://arxiv.org/abs/1802.01561 . - · arXiv
  36. [44][44] A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. D. Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Petersen, S. Legg, V. Mnih, K. Kavukcuoglu, and D. Silver. Massively parallel methods for deep reinforcement learning. CoRR , abs/1507.04296, 2015. URL [http://arxiv.org/abs/1507.04296 .](http://arxiv.org/abs/1507.04296) - · arXiv
  37. [45][45] D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver. Distributed Prioritized Experience Replay. In International Conference on Learning Representations , 2018. -
  38. [46][46] J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems , 2012. -

Cited by

Sorted by citations

Authors
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., & Levine, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Pages
651-673
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1806.10293
arXiv
1806.10293
OpenAlex
W2810785043
Cited by
697
References
4
Citations by year
2026: 2 · 2025: 16 · 2024: 58 · 2023: 56 · 2022: 91 · 2021: 203 · 2020: 169 · 2019: 91 · 2018: 11
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Conference on Robot Learning
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
Keywords
deep reinforcement learning, grasping, feedback control, Q-learning, self-supervised learning, regrasping, dynamic manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review