CoChat
Sign Up
Cited by 73Industrial Vision Systems and Defect DetectionNon-Destructive Testing TechniquesWelding Techniques and Residual Stresses

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

A Spatio-Temporal Ensemble Deep Learning Architecture for Real-Time Defect Detection during Laser Welding on Low Power Embedded Computing Boards

Open access
Knaak, Christian, von Eßen, Jakob, Kröger, Moritz, Schulze, Frederic, Abels, Peter, Gillner, Arnold·2021·Sensors
DOI 10.3390/s21124205PMC PMC8235065

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

In modern production environments, advanced and intelligent process monitoring strategies are required to enable an unambiguous diagnosis of the process situation and thus of the final component quality. In addition, the ability to recognize the current state of product quality in real-time is an important prerequisite for autonomous and self-improving manufacturing systems. To address these needs, this study investigates a novel ensemble deep learning architecture based on convolutional neural networks (CNN), gated recurrent units (GRU) combined with high-performance classification algorithms such as k-nearest neighbors (kNN) and support vector machines (SVM). The architecture uses spatio-temporal features extracted from infrared image sequences to locate critical welding defects including lack of fusion (false friends), sagging, lack of penetration, and geometric deviations of the weld seam. In order to evaluate the proposed architecture, this study investigates a comprehensive scheme based on classical machine learning methods using manual feature extraction and state-of-the-art deep learning algorithms. Optimal hyperparameters for each algorithm are determined by an extensive grid search. Additional work is conducted to investigate the significance of various geometrical, statistical and spatio-temporal features extracted from the keyhole and weld pool regions. The proposed method is finally validated on previously unknown welding trials, achieving the highest detection rates and the most robust weld defect recognition among all classification methods investigated in this work. Ultimately, the ensemble deep neural network is implemented and optimized to operate on low-power embedded computing devices with low latency (1.1 ms), demonstrating sufficient performance for real-time applications.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/s21124205

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Schematic overview of the evaluation process established in this work.
Figure 1 Schematic overview of the evaluation process established in this work.CC BY · PubMed Central
Figure 2 Proposed spatio-temporal ensemble deep neural network architecture based on convolutional layers, gated recurrent units (GRU) and different classification heads for weld defect detection.
Figure 2 Proposed spatio-temporal ensemble deep neural network architecture based on convolutional layers, gated recurrent units (GRU) and different classification heads for weld defect detection.CC BY · PubMed Central
Figure 3 (a) Photograph of the welding optics with coaxially integrated cameras; (b) Drawing of welding sheets with different slot sizes (middle sheet); (c) Side view of the sheet configuration used during the welding experiments; (d) Photograph (top view) of two welding trails (P = 3.3 KW, v = 50 mm/s, ds = 0.6 mm, Argon shield gas flow = 60 L/min).
Figure 3 (a) Photograph of the welding optics with coaxially integrated cameras; (b) Drawing of welding sheets with different slot sizes (middle sheet); (c) Side view of the sheet configuration used during the welding experiments; (d) Photograph (top view) of two welding trails (P = 3.3 KW, v = 50 mm/s, ds = 0.6 mm, Argon shield gas flow = 60 L/min).CC BY · PubMed Central
Figure 4 (a,b) Original image and geometrical features extracted from keyhole and weld pool regions. (c,d) Detected keyhole and weld pool contours (filled) based on two-step binarization of the original images.
Figure 4 (a,b) Original image and geometrical features extracted from keyhole and weld pool regions. (c,d) Detected keyhole and weld pool contours (filled) based on two-step binarization of the original images.CC BY · PubMed Central
Figure 5 Example of MWIR image data and sample distribution of different quality states based on 13 weld trials (14,530 samples) that form the welding data set.
Figure 5 Example of MWIR image data and sample distribution of different quality states based on 13 weld trials (14,530 samples) that form the welding data set.CC BY · PubMed Central
Figure 6 Photographs from different perspectives of the welding defects investigated in this study.
Figure 6 Photographs from different perspectives of the welding defects investigated in this study.CC BY · PubMed Central
Figure 7 The 20 most important features based on 172 geometrical and statistical characteristics of the weld pool, keyhole and overall image statistics extracted from MWIR images (starting with the left).
Figure 7 The 20 most important features based on 172 geometrical and statistical characteristics of the weld pool, keyhole and overall image statistics extracted from MWIR images (starting with the left).CC BY · PubMed Central
Figure 8 Layer activations based on 12 different filter kernels in the first layer of a CNN that was trained to recognize welding defects based on images from the MWIR camera. Each square shows a single activation map.
Figure 8 Layer activations based on 12 different filter kernels in the first layer of a CNN that was trained to recognize welding defects based on images from the MWIR camera. Each square shows a single activation map.CC BY · PubMed Central
Figure 9 Performance comparison of different conventional machine learning and deep learning classification methods. Optimal hyperparameter for each classifier were found via grid search (Table A3). The median scores are displayed in the top diagram.
Figure 9 Performance comparison of different conventional machine learning and deep learning classification methods. Optimal hyperparameter for each classifier were found via grid search (Table A3). The median scores are displayed in the top diagram.CC BY · PubMed Central
Figure 10 The metallographic characterization, the resulting ground truth data and the classification results for weld 42 based on the proposed ensemble CNN-GRU architecture.
Figure 10 The metallographic characterization, the resulting ground truth data and the classification results for weld 42 based on the proposed ensemble CNN-GRU architecture.CC BY · PubMed Central
Figure 11 The metallographic characterization, the resulting ground truth data and the classification results for weld 46 based on the proposed ensemble CNN-GRU architecture.
Figure 11 The metallographic characterization, the resulting ground truth data and the classification results for weld 46 based on the proposed ensemble CNN-GRU architecture.CC BY · PubMed Central
Figure 12 The metallographic characterization, the resulting ground truth data and the classification results for weld 48 based on the proposed ensemble CNN-GRU architecture.
Figure 12 The metallographic characterization, the resulting ground truth data and the classification results for weld 48 based on the proposed ensemble CNN-GRU architecture.CC BY · PubMed Central
Figure 13 The metallographic characterization, the resulting ground truth data and the classification results for weld 216 based on the proposed ensemble CNN-GRU architecture.
Figure 13 The metallographic characterization, the resulting ground truth data and the classification results for weld 216 based on the proposed ensemble CNN-GRU architecture.CC BY · PubMed Central
Figure 14 Inference results of the CNN-GRU architecture after optimization via TensorRT for different hardware setups.
Figure 14 Inference results of the CNN-GRU architecture after optimization via TensorRT for different hardware setups.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 61

  1. You, 2013, Review of laser welding monitoring, Sci. Technol. Weld. Join., 19, 181, 10.1179/1362171813Y.0000000180 · DOI
  2. Shao, 2005, Review of techniques for on-line monitoring and inspection of laser welding, J. Phys. Conf. Ser., 15, 101, 10.1088/1742-6596/15/1/017 · DOI
  3. Stavridis, 2017, Quality assessment in laser welding: A critical review, Int. J. Adv. Manuf. Technol., 94, 1825, 10.1007/s00170-017-0461-4 · DOI
  4. Kim, 2012, Coaxial monitoring of keyhole during Yb:YAG laser welding, Opt. Laser Technol., 44, 1874, 10.1016/j.optlastec.2012.02.025 · DOI
  5. Courtois, 2014, A complete model of keyhole and melt pool dynamics to analyze instabilities and collapse during laser welding, J. Laser Appl., 26, 042001, 10.2351/1.4886835 · DOI
  6. Saeed, 2007, Weld pool surface depth measurement using a calibrated camera and structured light, Meas. Sci. Technol., 18, 2570, 10.1088/0957-0233/18/8/033 · DOI
  7. Bertrand, 2000, Application of near infrared pyrometry for continuous Nd:YAG laser welding of stainless steel, Appl. Surf. Sci., 168, 182, 10.1016/S0169-4332(00)00586-9 · DOI
  8. Kong, 2012, Real-time monitoring of laser welding of galvanized high strength steel in lap joint configuration, Opt. Laser Technol., 44, 2186, 10.1016/j.optlastec.2012.03.003 · DOI
  9. Purtonen, 2014, Monitoring and Adaptive Control of Laser Processes, Phys. Proc., 56, 1218, 10.1016/j.phpro.2014.08.038 · DOI
  10. Zhang, 2013, Coaxial monitoring of the fibre laser lap welding of Zn-coated steel sheets using an auxiliary illuminant, Opt. Laser Technol., 50, 167, 10.1016/j.optlastec.2013.03.001 · DOI
  11. Knaak, 2019, Deep learning-based semantic segmentation for in-process monitoring in laser welding applications, Appl. Mach. Learn., 11139, 1113905
  12. Tenner, 2017, Analytical model of the laser welding of zinc-coated steel sheets by the aid of videography, J. Laser Appl., 29, 22411, 10.2351/1.4983236 · DOI
  13. Schmidt, 2008, Analysis of YAG laser lap-welding of zinc coated steel sheets, CIRP Ann., 57, 213, 10.1016/j.cirp.2008.03.043 · DOI
  14. Wuest, 2016, Machine learning in manufacturing: Advantages, challenges, and applications, Prod. Manuf. Res., 4, 23
  15. Xing, 2017, Quality assessment of resistance spot welding process based on dynamic resistance signal and random forest based, Int. J. Adv. Manuf. Technol., 94, 327, 10.1007/s00170-017-0889-6 · DOI
  16. Knaak, 2018, Machine learning as a comparative tool to determine the relevance of signal features in laser welding, Procedia CIRP, 74, 623, 10.1016/j.procir.2018.08.073 · DOI
  17. Jager, 2008, Principal Component Imagery for the Quality Monitoring of Dynamic Laser Welding Processes, IEEE Trans. Ind. Electron., 56, 1307, 10.1109/TIE.2008.2008339 · DOI
  18. You, 2015, WPD-PCA-Based Laser Welding Process Monitoring and Defects Diagnosis by Using FNN and SVM, IEEE Trans. Ind. Electron., 62, 628, 10.1109/TIE.2014.2319216 · DOI
  19. Schmidhuber, 2015, Deep learning in neural networks: An overview, Neural Netw., 61, 85, 10.1016/j.neunet.2014.09.003 · DOI · Paper page
  20. Hinton, 2012, Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups, IEEE Signal Process. Mag., 29, 82, 10.1109/MSP.2012.2205597 · DOI · Paper page
  21. Kruger, 2013, Deep Hierarchies in the Primate Visual Cortex: What Can We Learn for Computer Vision?, IEEE Trans. Pattern Anal. Mach. Intell., 35, 1847, 10.1109/TPAMI.2012.272 · DOI
  22. Mohamed, A.-R., Sainath, T.N., Dahl, G.E., Ramabhadran, B., Hinton, G.E., and Picheny, M.A. (2011, January 22–27). Deep Belief Networks using discriminative features for phone recognition. Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic. · DOI
  23. Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Buc, F.D., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates Inc.
  24. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2021, May 10). Caffe: Convolutional Architecture for Fast Feature Embedding. Available online: http://arxiv.org/pdf/1408.5093v1. · arXiv
  25. Wang, 2018, Deep learning for smart manufacturing: Methods and applications, J. Manuf. Syst., 48, 144, 10.1016/j.jmsy.2018.01.003 · DOI · Paper page
  26. Knaak, 2021, Improving Build Quality in Laser Powder Bed Fusion Using High Dynamic Range Imaging and Model-Based Reinforcement Learning, IEEE Access, 9, 55214, 10.1109/ACCESS.2021.3067302 · DOI
  27. Xue, B., Chang, B., and Du, D. (2021). Multi-Output Monitoring of High-Speed Laser Welding State Based on Deep Learning. Sensors, 21. · DOI
  28. Božič, A., Kos, M., and Jezeršek, M. (2020). Power Control during Remote Laser Welding Using a Convolutional Neural Network. Sensors, 20. · DOI
  29. Zhang, 2020, Real-time penetration state monitoring using convolutional neural network for laser welding of tailor rolled blanks, J. Manuf. Syst., 54, 348, 10.1016/j.jmsy.2020.01.006 · DOI
  30. Pilarski, 2014, First Steps towards an Intelligent Laser Welding Architecture Using Deep Neural Networks and Reinforcement Learning, Procedia Technol., 15, 474, 10.1016/j.protcy.2014.09.007 · DOI
  31. Pallas, 2019, A convolutional approach to quality monitoring for laser manufacturing, J. Intell. Manuf., 31, 789
  32. Liu, T., Bao, J., Wang, J., and Zhang, Y. (2018). A Hybrid CNN–LSTM Algorithm for Online Defect Recognition of CO2 Welding. Sensors, 18. · DOI
  33. Ouyang, 2019, A 3D-CNN and LSTM Based Multi-Task Learning Architecture for Action Recognition, IEEE Access, 7, 40757, 10.1109/ACCESS.2019.2906654 · DOI
  34. Fan, Y., Lu, X., Li, D., and Liu, Y. (2016, January 12–16). Video-based emotion recognition using CNN-RNN and C3D hybrid networks. Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan. · DOI · Paper page
  35. Valiente, R., Zaman, M., Ozer, S., and Fallah, Y.P. (2019, January 9–12). Controlling Steering Angle for Cooperative Self-driving Vehicles utilizing CNN and LSTM-based Deep Networks. Proceedings of the 30th IEEE Intelligent Vehicles Symposium, Paris, France. · DOI
  36. Yin, W., Kann, K., Yu, M., and Schütze, H. (2021, May 12). Comparative Study of CNN and RNN for Natural Language Processing. Available online: http://arxiv.org/pdf/1702.01923v1. · arXiv
  37. He, K., Zhang, X., Ren, S., and Sun, J. (2021, May 12). Identity Mappings in Deep Residual Networks. Available online: https://arxiv.org/pdf/1603.05027. · arXiv
  38. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L. (2018, January 18–23). Mobilenetv2: Inverted Residuals and Linear Bottlenecks. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA. · DOI · Paper page
  39. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2021, May 11). Rethinking the Inception Architecture for Computer Vision. Available online: https://arxiv.org/pdf/1512.00567. · arXiv
  40. Bishop, C.M. (2006). Pattern recognition and machine learning. Information Science and Statistics, Springer.
  41. Nixon, M.S., and Aguado, A.S. (2012). Feature Extraction & Image Processing for Computer Vision, Academic Press. [3rd ed.].
  42. Runkler, T.A. (2012). Data Analytics: Models and Algorithms for Intelligent Data Analysis, Vieweg+Teubner Verlag. · DOI
  43. Pedregosa, 2011, Scikit-learn: Machine Learning in Python, J. Mach. Learn. Res., 12, 2825
  44. Lee, 2017, A Convolutional Neural Network for Fault Classification and Diagnosis in Semiconductor Manufacturing Processes, IEEE Trans. Semicond. Manuf., 30, 135, 10.1109/TSM.2017.2676245 · DOI · Paper page
  45. Arif, S., Wang, J., Hassan, T.U., and Fei, Z. (2019). 3D-CNN-Based Fused Feature Maps with LSTM Applied to Action Recognition. Future Internet, 11. · DOI
  46. Kim, D., Cho, H., Shin, H., Lim, S.-C., and Hwang, W. (2019). An Efficient Three-Dimensional Convolutional Neural Network for Inferring Physical Interaction Force from Video. Sensors, 19. · DOI
  47. Zhu, 2017, Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM, IEEE Access, 5, 4517, 10.1109/ACCESS.2017.2684186 · DOI
  48. Ullah, 2018, Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features, IEEE Access, 6, 1155, 10.1109/ACCESS.2017.2778011 · DOI
  49. Rana, R. (2021, May 11). Gated Recurrent Unit (GRU) for Emotion Classification from Noisy Speech. Available online: http://arxiv.org/pdf/1612.07778v1. · arXiv
  50. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv.
  51. Ganaie, M.A., Hu, M., Tanveer, M., and Suganthan, P.N. (2021, May 11). Ensemble Deep Learning: A Review. Available online: https://arxiv.org/pdf/2104.02395. · arXiv
  52. DeWitt, D.P., and Nutter, G.D. (1988). Theory and Practice of Radiation Thermometry, John Wiley and Sons. · DOI
  53. Suzuki, 1985, Topological structural analysis of digitized binary images by border following, Comput. Vis. Graph. Image Process., 30, 32, 10.1016/0734-189X(85)90016-7 · DOI
  54. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, The MIT Press.
  55. Sun, C., Shrivastava, A., Singh, S., and Gupta, A. (2017, January 22–29). Revisiting unreasonable effectiveness of data in deep learning era. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy. · DOI · Paper page
  56. Zongker, D., and Jain, A. (1996, January 29). Algorithms for feature selection: An evaluation. Proceedings of the 13th International Conference on Pattern Recognition, Vienna, Austria. · DOI
  57. Kotsiantis, 2005, Handling imbalanced datasets: A review, GESTS Int. Trans. Comput. Sci. Eng., 30, 25
  58. Chawla, 2002, SMOTE: Synthetic minority over-sampling technique, J. Artif. Intell. Res., 16, 321, 10.1613/jair.953 · DOI · Paper page
  59. Bifet, A., and Gavaldà, R. (2007, January 26–28). Learning from Time-Changing Data with Adaptive Windowing. Proceedings of the 2007 SIAM International Conference on Data Mining, Minneapolis, MN, USA. · DOI
  60. Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., and Zhang, G. (2018). Learning under Concept Drift: A Review. IEEE Trans. Knowl. Data Eng. · DOI · Paper page
  61. Fitzgibbon, A.W., and Fisher, R.B. (1995, January 11–14). A Buyer’s Guide to Conic Fitting. Proceedings of the 6th British Machine Vision Conference, Birmingham, UK. · DOI

Metadata

Authors
Knaak, C., von Eßen, J., Kröger, M., Schulze, F., Abels, P., & Gillner, A.
Affiliations
  • Fraunhofer Institute for Laser Technology (DE)
  • RWTH Aachen University (DE)
Corresponding author
Knaak, Christian
ORCID
Venue
Sensors
Publisher
MDPI AG
Published
2021-06-18
Volume
21
Issue
12
Pages
4205
ISSN
1424-8220 (ISSN-L)
DOI
10.3390/s21124205
PMC
PMC8235065
OpenAlex
W3173506310
Cited by
73
References
5
Citations by year
2026: 9 · 2025: 14 · 2024: 18 · 2023: 19 · 2022: 10 · 2021: 3
FWCI
6.47
Type
journal article
Language
en
Open access
Licence
Locations
  • Sensors (publishedVersion, cc-by)
  • Preprints.org (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
  • RWTH Publications (RWTH Aachen)
Funders
H2020 Leadership in Enabling and Industrial Technologies (825030; 637081)
Topics
  • Physical Sciences › Engineering › Industrial and Manufacturing Engineering › Industrial Vision Systems and Defect Detection
  • Physical Sciences › Engineering › Mechanical Engineering › Non-Destructive Testing Techniques
  • Physical Sciences › Engineering › Mechanical Engineering › Welding Techniques and Residual Stresses
Keywords
laser welding, weld defect detection, spatio-temporal features, ensemble deep learning, convolutional neural networks, gated recurrent unit, infrared imaging, lack of fusion, lack of penetration, sagging, weld bead geometry, real-time classification, process monitoring, hyperparameter optimization
MeSH terms
Deep Learning, Algorithms, Lasers, Welding, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review