A deep learning-enhanced Digital Twin framework for improving safety and reliability in human–robot collaborative manufacturing
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
In Industry 5.0, Digital Twins bring in flexibility and efficiency for smart manufacturing. Recently, the success of artificial intelligence techniques such as deep learning has led to their adoption in manufacturing and especially in human–robot collaboration. Collaborative manufacturing tasks involving human operators and robots pose significant safety and reliability concerns. In response to these concerns, a deep learning-enhanced Digital Twin framework is introduced through which human operators and robots can be detected and their actions can be classified during the manufacturing process, enabling autonomous decision making by the robot control system. Developed using Unreal Engine 4, our Digital Twin framework complies with the Robotics Operating System specification, and supports synchronous control and communication between the Digital Twin and the physical system. In our framework, a fully-supervised detector based on a faster region-based convolutional neural network is firstly trained on synthetic data generated by the Digital Twin, and then tested on the physical system to demonstrate the effectiveness of the proposed Digital Twin-based framework. To ensure safety and reliability, a semi-supervised detector is further designed to bridge the gap between the twin system and the physical system, and improved performance is achieved by the semi-supervised detector compared to the fully-supervised detector that is simply trained on either synthetic data or real data. The evaluation of the framework in multiple scenarios in which human operators collaborate with a Universal Robot 10 shows that it can accurately detect the human and robot, and classify their actions under a variety of conditions. The data from this evaluation have been made publicly available, and can be widely used for research and operational purposes. Additionally, a semi-automated annotation tool from the Digital Twin framework is published to benefit the collaborative robotics community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Self-citationThe Pascal Visual Object Classes Challenge: A Retrospective
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0733-5 · Cited by 7,139
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980
Digital Twin in Industry: State-of-the-Art
Tao, F., Zhang, H., Liu, A., & Nee, A. Y. C. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2873186 · Cited by 5,871
Simple online and realtime tracking
Bewley, A., Ge, Z., Ott, L., Ramos, F., & Upcroft, B. (2016) · 2016 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2016.7533003 · Cited by 4,433
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., & Raffel, C. (2020) · arXiv · doi:10.48550/arxiv.2001.07685 · Cited by 2,892
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579
Efficient Non-Maximum Suppression
Neubeck, A., & Van Gool, L. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.479 · Cited by 2,118
Playing for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,815
A Survey on Performance Metrics for Object-Detection Algorithms
Padilla, R., Netto, S. L., & da Silva, E. A. B. (2020) · 2020 International Conference on Systems, Signals and Image Processing (IWSSIP) · doi:10.1109/iwssip48289.2020.9145130 · Cited by 1,605
Enabling technologies and tools for digital twin
Qi, Q., Tao, F., Hu, T., Anwer, N., Liu, A., Wei, Y., Wang, L., & Nee, A. Y. C. (2021) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2019.10.001 · Cited by 1,579
Review of digital twin applications in manufacturing
Cimino, C., Negri, E., & Fumagalli, L. (2019) · Computers in Industry · doi:10.1016/j.compind.2019.103130 · Cited by 874
A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit
Padilla, R., Passos, W. L., Dias, T. L. B., Netto, S. L., & da Silva, E. A. B. (2021) · Electronics · doi:10.3390/electronics10030279 · Cited by 735
Human–Robot Collaboration in Manufacturing Applications: A Review
Matheson, E., Minto, R., Zampieri, E. G. G., Faccio, M., & Rosati, G. (2019) · Robotics · doi:10.3390/robotics8040100 · Cited by 668
Unity: A General Platform for Intelligent Agents
Juliani, A., Berges, V.-P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.02627 · Cited by 626
CAD2RL: Real Single-Image Flight Without a Single Real Image
Sadeghi, F., & Levine, S. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.034 · Cited by 618
Rathore, M. M., Shah, S. A., Shukla, D., Bentafat, E., & Bakiras, S. (2021) · IEEE Access · doi:10.1109/access.2021.3060863 · Cited by 616
Digital Twins: State of the art theory and practice, challenges, and open research questions
Sharma, A., Kosasih, E., Zhang, J., Brintrup, A., & Calinescu, A. (2022) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2022.100383 · Cited by 587
Self-citationA survey on AI-driven digital twins in Industry 4.0: smart manufacturing and advanced robotics
Huang, Z., Shen, Y., Li, J., Fey, M., & Brecher, C. (2021) · Sensors · doi:10.3390/s21196340 · Cited by 418
Honig, S., & Oron-Gilad, T. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00861 · Cited by 338
A Survey of Uncertainty in Deep Neural Networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03342 · Cited by 232
Modeling of Cloud-Based Digital Twins for Smart Manufacturing with MT Connect
Hu, L., Nguyen, N.-T., Tao, W., Leu, M. C., Liu, X. F., Shahriar, M. R., & Al Sunny, S. M. N. (2018) · Procedia Manufacturing · doi:10.1016/j.promfg.2018.07.155 · Cited by 219
A survey of Digital Twin techniques in smart manufacturing and management of energy applications
Wang, Y., Kang, X., & Chen, Z. (2022) · Green Energy and Intelligent Transportation · doi:10.1016/j.geits.2022.100014 · Cited by 147
Digital twin for integration of design-manufacturing-maintenance: an overview
Fu, Y., Zhu, G., Zhu, M., & Xuan, F. (2022) · Chinese Journal of Mechanical Engineering · doi:10.1186/s10033-022-00760-x · Cited by 98
Cited by
Digital Twins for Healthcare Using Wearables
Johnson, Z., & Saikia, M. J. (2024) · Bioengineering · doi:10.3390/bioengineering11060606 · Cited by 74
Dini, P., & Paolini, D. (2025) · Batteries · doi:10.3390/batteries11030107 · Cited by 40
“…neural networks has demonstrated considerable success not only in the electric vehicle industry but also across various applications, such as energy efficiency monitoring in luxury yachts, hotel energy management, autonomous navigation for automotive vehicles, and other robotic systems [32–35].”
Liu, X., Li, G., Xiang, F., Tao, B., & Jiang, G. (2024) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2024.09.006 · Cited by 18
Artificial Intelligence in Electric Vehicle Battery Disassembly: A Systematic Review
Ai, Z., Nee, A. Y. C., & Ong, S. K. (2024) · Automation · doi:10.3390/automation5040028 · Cited by 16
Metadata
- Authors
- Wang, S., Zhang, J., Wang, P., Law, J., Calinescu, R., & Mihaylova, L.
- Venue
- Robotics and Computer-Integrated Manufacturing
- Publisher
- Elsevier BV
- Published
- 2024-02
- Volume
- 85
- Pages
- 102608
- ISSN
- 0736-5845 (ISSN-L)
- OpenAlex
- W4384472141
- Cited by
- 169
- References
- 27
- FWCI
- 18.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.