CoChat

Robot learning towards smart robotic manufacturing: A review

Open access
Liu, Zhihao, Liu, Quan, Xu, Wenjun, Wang, Lihui, Zhou, Zude · 2022 · Robotics and Computer-Integrated Manufacturing
Cited by 215

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Abstract

Robotic equipment has been playing a central role since the proposal of smart manufacturing. Since the beginning of the first integration of industrial robots into production lines, industrial robots have enhanced productivity and relieved humans from heavy workloads significantly. Towards the next generation of manufacturing, this review first introduces the comprehensive background of smart robotic manufacturing within robotics, machine learning, and robot learning. Definitions and categories of robot learning are summarised. Concretely, imitation learning, policy gradient learning, value function learning, actor-critic learning, and model-based learning as the leading technologies in robot learning are reviewed. Training tools, benchmarks, and comparisons amongst different robot learning methods are delivered. Typical industrial applications in robotic grasping, assembly, process control, and industrial human-robot collaboration are listed and discussed. Finally, open problems and future research directions are summarised.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.rcim.2022.102360

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 81

  1. Kang, 2016, Smart manufacturing: past research, present findings, and future directions, Int. J. Precision Eng. Manuf.-Green Technol., 3, 111, 10.1007/s40684-016-0015-5 · DOI · Paper page
  2. Wang, 2019, From intelligence science to intelligent manufacturing, Engineering, 5, 615, 10.1016/j.eng.2019.04.011 · DOI · Paper page
  3. Kaelbling, 2020, The foundation of efficient robot learning, Science, 369, 915, 10.1126/science.aaz7597 · DOI
  4. Cui, 2021, Toward next-generation learned robot manipulation, Sci. Robot., 6, 10.1126/scirobotics.abd9461 · DOI
  5. Kroemer, 2021, A review of robot learning for manipulation: challenges, representations, and algorithms, J. Mach. Learn. Res., 22
  6. Xie, 2020, Robot learning from demonstration for path planning: a review, Sci. China: Technol. Sci., 63, 1
  7. Ravichandar, 2020, Recent advances in robot learning from demonstration, Ann. Rev. Control, Robot. Autonomous Syst., 3, 297, 10.1146/annurev-control-100819-063206 · DOI · Paper page
  8. Wang, 2019, Learning and comfort in human–robot interaction: a review, Appl. Sci., 9, 5152, 10.3390/app9235152 · DOI
  9. Zhu, 2018, Robot learning from demonstration in robotic assembly: a survey, Robotics, 7, 17, 10.3390/robotics7020017 · DOI · Paper page
  10. Digani, 2015, Ensemble coordination approach in multi-agv systems applied to industrial warehouses, IEEE Trans. Autom. Sci. Eng., 12, 922, 10.1109/TASE.2015.2446614 · DOI
  11. Liu, 2019, Human-robot collaboration in disassembly for sustainable manufacturing, Int. J. Prod. Res., 57, 4027, 10.1080/00207543.2019.1578906 · DOI · Paper page
  12. Lu, 2020, Smart manufacturing process and system automation–a critical review of the standards and envisioned scenarios, J. Manuf. Syst., 56, 312, 10.1016/j.jmsy.2020.06.010 · DOI · Paper page
  13. Qu, 2019, Smart manufacturing systems: state of the art and future trends, Int. J. Adv. Manuf. Technol., 103, 3751, 10.1007/s00170-019-03754-7 · DOI · Paper page
  14. Wang, 2019, Symbiotic human-robot collaborative assembly, CIRP Ann. Manuf. Technol., 68, 701, 10.1016/j.cirp.2019.05.002 · DOI · Paper page
  15. Wang, 2021, Evaluation and optimization of dual-arm robot path planning for human–robot collaborative tasks in smart manufacturing contexts, ASME Lett. Dyn. Syst. Control, 1
  16. Liu, 2021, Sensorless haptic control for human-robot collaborative assembly, CIRP J. Manuf. Sci. Technol., 32, 132, 10.1016/j.cirpj.2020.11.015 · doi:10.1016/j.cirpj.2020.11.015
  17. Al-Yacoub, 2021, Improving human robot collaboration through force/torque based learning for object manipulation, Rob. Comput. Integr. Manuf., 69, 10.1016/j.rcim.2020.102111 · DOI
  18. Liu, 2021, Sensorless force estimation for industrial robots using disturbance observer and neural learning of friction approximation, Rob. Comput. Integr. Manuf., 71, 10.1016/j.rcim.2021.102168 · doi:10.1016/j.rcim.2021.102168
  19. Liu, 2020, Dynamic risk assessment and active response strategy for industrial human-robot collaboration, Comput. Ind. Eng., 141, 10.1016/j.cie.2020.106302 · DOI
  20. Liu, 2021, Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function, Adv. Eng. Inf., 49, 10.1016/j.aei.2021.101360 · DOI
  21. Liu, 2018, Human-robot collaborative manufacturing using cooperative game: framework and implementation, Procedia CIRP, 72, 87, 10.1016/j.procir.2018.03.172 · DOI
  22. Billard, 2019, Trends and challenges in robot manipulation, Science, 364, 10.1126/science.aat8414 · DOI · Paper page
  23. Cheng, 2019, Purposive learning: robot reasoning about the meanings of human activities, Sci. Robot., 4, 10.1126/scirobotics.aav1530 · DOI
  24. Mahler, 2019, Learning ambidextrous robot grasping policies, Sci. Robot., 4, 10.1126/scirobotics.aau4984 · DOI · Paper page
  25. Ichnowski, 2020, Deep learning can accelerate grasp-optimized motion planning, Sci. Robot., 5, 10.1126/scirobotics.abd7710 · DOI
  26. Sundaram, 2020, Robots learn to identify objects by feeling, Sci. Robot., 5, 10.1126/scirobotics.abf1502 · DOI
  27. Chernova, 2014, Robot learning from human teachers, Synth. Lect. Artif. Intell. Mach. Learn., 8, 1
  28. Won, 2020, An adaptive deep reinforcement learning framework enables curling robots with human-like performance in real-world conditions, Sci. Robot., 5, 10.1126/scirobotics.abb9764 · DOI
  29. Lazaro-Gredilla, 2019, Beyond imitation: zero-shot task transfer on robots by learning concepts as cognitive programs, Sci. Robot., 4, 10.1126/scirobotics.aav3150 · DOI · Paper page
  30. Pedersen, 2016, Robot skills for manufacturing: from concept to industrial deployment, Rob. Comput. Integr. Manuf., 37, 282, 10.1016/j.rcim.2015.04.002 · DOI · Paper page
  31. Bemporad, 2002, The explicit linear quadratic regulator for constrained systems, Automatica, 38, 3, 10.1016/S0005-1098(01)00174-1 · DOI · Paper page
  32. M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, et al., End to end learning for self-driving cars, arXiv preprint arXiv:160407316. (2016).
  33. Williams, 1992, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Mach. Learn., 8, 229, 10.1023/A:1022672621406 · DOI · Paper page
  34. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, arXiv preprint arXiv:170706347. (2017).
  35. D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, et al., Qt-opt: scalable deep reinforcement learning for vision-based robotic manipulation, arXiv preprint arXiv:180610293. (2018).
  36. Yao, 2018, Sensor-less external force detection for industrial manipulators to facilitate physical human-robot interaction, J. Mech. Sci. Technol., 32, 4909, 10.1007/s12206-018-0939-5 · doi:10.1007/s12206-018-0939-5
  37. De Boer, 2005, A tutorial on the cross-entropy method, Ann. Oper. Res., 134, 19, 10.1007/s10479-005-5724-z · DOI · Paper page
  38. Levine, 2016, End-to-end training of deep visuomotor policies, J. Mach. Learn. Res., 17, 1334
  39. Bristow, 2006, A survey of iterative learning control, IEEE Control Syst. Mag., 26, 96, 10.1109/MCS.2006.1636313 · DOI · Paper page
  40. James, 2020, Rlbench: the robot learning benchmark & learning environment, IEEE Robot. Autom. Lett., 5, 3019, 10.1109/LRA.2020.2974707 · DOI · Paper page
  41. Zhang, 2021, Digital twin-enabled reconfigurable modeling for smart manufacturing systems, Int. J. Computer Integr. Manuf., 34, 709, 10.1080/0951192X.2019.1699256 · DOI
  42. de Souza, 2021, Robotic grasping: from wrench space heuristics to deep learning policies, Rob. Comput. Integr. Manuf., 71, 10.1016/j.rcim.2021.102176 · DOI
  43. Moosmann, 2021, Separating entangled workpieces in random bin picking using deep reinforcement learning, Procedia CIRP, 104, 881, 10.1016/j.procir.2021.11.148 · DOI
  44. L. Berscheid, C. Friedrich, T. Kröger, Robot learning of 6 dof grasping using model-based adaptive primitives, arXiv preprint arXiv:210312810. (2021). · DOI
  45. Mohammed, 2020, Pick and place objects in a cluttered scene using deep reinforcement learning, Int. J. Mech. Mechatron. Eng., 20, 50
  46. Wang, 2021, A visual grasping strategy for improving assembly efficiency based on deep reinforcement learning, J. Sens., 2021
  47. Akkaladevi, 2019, Towards reinforcement based learning of an assembly process for human robot collaboration, Procedia Manuf., 38, 1491, 10.1016/j.promfg.2020.01.138 · DOI
  48. De Winter, 2019, Accelerating interactive reinforcement learning by human advice for an assembly task by a cobot, Robotics, 8, 104, 10.3390/robotics8040104 · DOI
  49. Duque, 2019, Trajectory generation for robotic assembly operations using learning by demonstration, Rob. Comput. Integr. Manuf., 57, 292, 10.1016/j.rcim.2018.12.007 · DOI
  50. Kim, 2020, Reinforcement learning based on movement primitives for contact tasks, Rob. Comput. Integr. Manuf., 62, 10.1016/j.rcim.2019.101863 · DOI
  51. Kyrarini, 2019, Robot learning of industrial assembly task via human demonstrations, Autonom. Rob., 43, 239, 10.1007/s10514-018-9725-6 · DOI
  52. Li, 2019, Robot skill acquisition in assembly process using deep reinforcement learning, Neurocomputing, 345, 92, 10.1016/j.neucom.2019.01.087 · DOI
  53. Kristensen, 2019, Towards a robot simulation framework for e-waste disassembly using reinforcement learning, Procedia Manuf., 38, 225, 10.1016/j.promfg.2020.01.030 · doi:10.1016/j.promfg.2020.01.030
  54. Andersen, 2019, Self-learning processes in smart factories: deep reinforcement learning for process control of robot brine injection, Procedia Manuf., 38, 171, 10.1016/j.promfg.2020.01.023 · doi:10.1016/j.promfg.2020.01.023
  55. Brito, 2020, A machine learning approach for collaborative robot smart manufacturing inspection for quality control systems, Procedia Manuf., 51, 11, 10.1016/j.promfg.2020.10.003 · DOI · Paper page
  56. Duguleana, 2012, Obstacle avoidance of redundant manipulators using neural networks based reinforcement learning, Rob. Comput. Integr. Manuf., 28, 132
  57. M.S.A. Hameed, M.M. Khan, A. Schwung, Curiosity based reinforcement learning on robot manufacturing cell, arXiv preprint arXiv:201108743. (2020).
  58. Imtiaz, 2021, A comparison of two reinforcement learning algorithms for robotic pick and place with non-visual sensing, Int. J. Mech. Eng. Robot. Res., 10, 526
  59. Jaradat, 2011, Reinforcement based mobile robot navigation in dynamic environment, Rob. Comput. Integr. Manuf., 27, 135, 10.1016/j.rcim.2010.06.019 · DOI · Paper page
  60. Luo, 2019, A robot learning method with physiological interface for teleoperation systems, Appl. Sci., 9, 2099, 10.3390/app9102099 · DOI
  61. Maldonado-Ramirez, 2021, A visual path-following learning approach for industrial robots using drl, Rob. Comput. Integr. Manuf., 71, 10.1016/j.rcim.2021.102130 · DOI
  62. Tsai, 2020, Utilization of a reinforcement learning algorithm for the accurate alignment of a robotic arm in a complete soft fabric shoe tongues automation process, J. Manuf. Syst., 56, 501, 10.1016/j.jmsy.2020.07.001 · DOI
  63. Wang, 2020, Learning mobile manipulation through deep reinforcement learning, Sensors, 20, 939, 10.3390/s20030939 · DOI · Paper page
  64. Wang, 2015, Nonparametric statistical learning control of robot manipulators for trajectory or contour tracking, Rob. Comput. Integr. Manuf., 35, 96, 10.1016/j.rcim.2015.03.002 · DOI
  65. Ye, 2020, Bagging for gaussian mixture regression in robot learning from demonstration, J. Intell. Manuf., 1
  66. Zeng, 2018, Robot learning human stiffness regulation for hybrid manufacture, Assembly Autom., 38, 539, 10.1108/AA-02-2018-019 · DOI
  67. Zhang, 2021, Robot learning system based on dynamic movement primitives and neural network, Neurocomputing, 451, 205, 10.1016/j.neucom.2021.04.034 · doi:10.1016/j.neucom.2021.04.034
  68. Liu, 2021, Task-level decision-making for dynamic and stochastic human-robot collaboration based on dual agents deep reinforcement learning, Int. J. Adv. Manuf. Technol., 115, 3533, 10.1007/s00170-021-07265-2 · DOI
  69. Oliff, 2020, Reinforcement learning for facilitating human-robot-interaction in manufacturing, J. Manuf. Syst., 56, 326, 10.1016/j.jmsy.2020.06.018 · DOI
  70. Rozo, 2016, Learning physical collaborative robot behaviors from human demonstrations, IEEE Trans. Rob., 32, 513, 10.1109/TRO.2016.2540623 · DOI · Paper page
  71. Sun, 2020, Learn how to assist humans through human teaching and robot learning in human-robot collaborative assembly, IEEE Trans. Syst., Man, Cybern.: Systems
  72. Wang, 2018, Facilitating human–robot collaborative tasks by teaching-learning-collaboration from human demonstrations, IEEE Trans. Autom. Sci. Eng., 16, 640, 10.1109/TASE.2018.2840345 · DOI · Paper page
  73. Wang, 2021, Predicting human intentions in human-robot and-over tasks through multimodal learning, IEEE Trans. Autom. Sci. Eng., 1
  74. Wang, 2021, Optimised learning from demonstrations for collaborative robots, Rob. Comput. Integr. Manuf., 71, 10.1016/j.rcim.2021.102169 · DOI
  75. Yu, 2020, Mastering the working sequence in human-robot collaborative assembly based on reinforcement learning, IEEE Access, 8, 163868, 10.1109/ACCESS.2020.3021904 · DOI
  76. Zhang, 2022, A reinforcement learning method for human-robot collaboration in assembly tasks, Rob. Comput. Integr. Manuf., 73, 10.1016/j.rcim.2021.102227 · doi:10.1016/j.rcim.2021.102227
  77. Leottau, 2018, Decentralized reinforcement learning of robot behaviors, Artif. Intell., 256, 130, 10.1016/j.artint.2017.12.001 · DOI
  78. Xu, 2021, Digital twin-based industrial cloud robotics: framework, control approach and implementation, J. Manuf. Syst., 58, 196, 10.1016/j.jmsy.2020.07.013 · DOI · Paper page
  79. de Bruin, 2018, Experience selection in deep reinforcement learning for control, J. Mach. Learn. Res., 19
  80. T.Z. Zhao, J. Luo, O. Sushkov, R. Pevceviciute, N. Heess, J. Scholz, et al., Offline meta-reinforcement learning for industrial insertion, arXiv preprint arXiv:211004276. (2021). · DOI
  81. Bogert, 2018, Multi-robot inverse reinforcement learning under occlusion with estimation of state transitions, Artif. Intell., 263, 46, 10.1016/j.artint.2018.07.002 · DOI

Cited by

See all 215

Metadata

Authors
Liu, Z., Liu, Q., Xu, W., Wang, L., & Zhou, Z.
Venue
Robotics and Computer-Integrated Manufacturing
Publisher
Elsevier BV
Published
2022-10
Volume
77
Pages
102360
ISSN
0736-5845 (ISSN-L)
OpenAlex
W4224221644
Cited by
215
References
33
FWCI
14.38
Type
journal article
Language
en
Open access
hybrid
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.