CoChat
Sign Up
PreprintCited by 47Human Pose and Action RecognitionContext-Aware Activity Recognition SystemsHand Gesture Recognition Systems

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Fine-Grained Activity Classification in Assembly Based on Multi-Visual Modalities

Open access
Chen, Haodong, Zendehdel, Niloofar, Leu, Ming C., Yin, Zhaozheng·2023·Journal of Intelligent Manufacturing
DOI 10.21203/rs.3.rs-2387286/v2

Preprint — published as 10.1007/s10845-023-02152-x · Paper page

Open this paper in CoChat — free account

Abstract

Assembly activity recognition and prediction help to improve productivity, quality control, and safety measures in smart factories. This study aims to sense, recognize, and predict a worker's continuous fine-grained assembly activities in a manufacturing platform. We propose a two-stage network for workers' fine-grained activity classification by leveraging scene-level and temporal-level activity features. The first stage is a feature awareness block that extracts scene-level features from multi-visual modalities, including red-green-blue (RGB) and hand skeleton frames. We use the transfer learning method in the first stage and compare three different pre-trained feature extraction models. Then, we transmit the feature information from the first stage to the second stage to learn the temporal-level features of activities. The second stage consists of the Recurrent Neural Network (RNN) layers and a final classifier. We compare the performance of two different RNNs in the second stage, including the Long Short-Term Memory (LSTM) and the Gated Recurrent Unit (GRU). The partial video observation method is used in the prediction of fine-grained activities. In the experiments using the trimmed activity videos, our model achieves an accuracy of > 99% on our dataset and > 98% on the public dataset UCF 101, outperforming the state-of-the-art models. The prediction model achieves an accuracy of > 97% in predicting activity labels using 50% of the onset activity video information. In the experiments using an untrimmed video with continuous assembly activities, we combine our recognition and prediction models and achieve an accuracy of > 91% in real time, surpassing the state-of-the-art models for the recognition of continuous assembly activities.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21203/rs.3.rs-2387286/v2

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z. and Liu, Y., 2021. Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities. ACM Computing Surveys (CSUR), 54(4), pp.1–40. https://doi.org/10.1145/3447744 · DOI · Paper page
  2. Rude, D.J., Adams, S. and Beling, P.A., 2018. Task recognition from joint tracking data in an operational manufacturing cell. Journal of Intelligent Manufacturing, 29(6), pp.1203–1217. https://doi.org/10.1007/s10845-015-1168-8 · DOI
  3. Xiao, J., Jing, L., Zhang, L., He, J., She, Q., Zhou, Z., Yuille, A. and Li, Y., 2022. Learning from temporal gradient for semi-supervised action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3252–3262. https://doi.org/10.48550/arXiv.2111.13241 · DOI
  4. Ahn, D., Kim, S., Hong, H. and Ko, B.C., 2023. STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3330–3339. https://doi.org/10.48550/arXiv.2210.07503 · DOI
  5. Chen, H., Leu, M.C. and Yin, Z., 2022. Real-Time Multi-modal Human-Robot Collaboration Using Gestures and Speech. Journal of Manufacturing Science and Engineering, pp.1–22. https://doi.org/10.1115/1.4054297 · DOI
  6. Chen, H., Leu, M.C., Tao, W. and Yin, Z., 2020, November. Design of a real-time human-robot collaboration system using dynamic gestures. In ASME International Mechanical Engineering Congress and Exposition (Vol. 84492, p. V02BT02A051). American Society of Mechanical Engineers. https://doi.org/10.1115/IMECE2020-23650 · DOI
  7. Al-Amin, M., Qin, R., Moniruzzaman, M., Yin, Z., Tao, W. and Leu, M.C., 2021. An individualized system of skeletal data-based CNN classifiers for action recognition in manufacturing assembly. Journal of Intelligent Manufacturing, pp.1–17. https://doi.org/10.1007/s10845-021-01815-x · DOI
  8. Tao, W., Al-Amin, M., Chen, H., Leu, M.C., Yin, Z. and Qin, R., 2020. Real-time assembly operation recognition with fog computing and transfer learning for human-centered intelligent manufacturing. Procedia Manufacturing, 48, pp.926–931. https://doi.org/10.1016/j.promfg.2020.05.131 · DOI
  9. Sherafat, B., Ahn, C.R., Akhavian, R., Behzadan, A.H., Golparvar-Fard, M., Kim, H., Lee, Y.C., Rashidi, A. and Azar, E.R., 2020. Automated methods for activity recognition of construction workers and equipment: State-of-the-art review. Journal of Construction Engineering and Management, 146(6), p.03120002. https://ascelibrary.org/doi/full/10.1061/(ASCE)CO.1943-7862.0001843 · DOI
  10. Zheng, T., Ardolino, M., Bacchetti, A. and Perona, M., 2021. The applications of Industry 4.0 technologies in manufacturing context: a systematic literature review. International Journal of Production Research, 59(6), pp.1922–1954. https://doi.org/10.1080/00207543.2020.1824085 · DOI
  11. Hu, Z., Yu, T., Zhang, Y. and Pan, S., 2020, September. Fine-grained activities recognition with coarse-grained labeled multi-modal data. In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers (pp. 644–649). https://doi.org/10.1145/3410530.3414320 · DOI
  12. Schuldt, C., Laptev, I. and Caputo, B., 2004, August. Recognizing human actions: a local SVM approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004. (Vol. 3, pp. 32–36). IEEE. https://doi.org/10.1109/ICPR.2004.1334462 · DOI
  13. Ryoo, M.S. and Aggarwal, J.K., 2009, September. Spatio-temporal relationship match: Video structure comparison for recognition of complex human activities. In 2009 IEEE 12th international conference on computer vision (pp. 1593–1600). IEEE. https://doi.org/10.1109/ICCV.2009.5459361 · DOI
  14. Xia, L., Chen, C.C. and Aggarwal, J.K., 2012, June. View invariant human action recognition using histograms of 3d joints. In 2012 IEEE computer society conference on computer vision and pattern recognition workshops (pp. 20–27). IEEE. https://doi.org/10.1109/CVPRW.2012.6239233 · DOI
  15. Soomro, K., Zamir, A.R. and Shah, M., 2012. UCF101: A dataset of 101 human action classes from videos in the wild. arXiv preprint arXiv:1212.0402. https://doi.org/10.48550/arXiv.1212.0402 · DOI · Paper page
  16. Rohrbach, M., Amin, S., Andriluka, M. and Schiele, B., 2012, June. A database for fine-grained activity detection of cooking activities. In 2012 IEEE conference on computer vision and pattern recognition (pp. 1194–1201). IEEE. https://doi.org/10.1109/CVPR.2012.6247801 · DOI · Paper page
  17. Byrne, J., Castañón, G., Li, Z. and Ettinger, G., 2023. Fine-grained Activities of People Worldwide. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 3308–3319). https://doi.org/10.48550/arXiv.2207.05182 · DOI
  18. Singh, B., Marks, T.K., Jones, M., Tuzel, O. and Shao, M., 2016. A multi-stream bi-directional recurrent neural network for fine-grained action detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1961–1970). https://doi.org/10.1109/CVPR.2016.216 · DOI · Paper page
  19. Pan, S., Berges, M., Rodakowski, J., Zhang, P. and Noh, H.Y., 2020. Fine-Grained Activity of Daily Living (ADL) Recognition Through Heterogeneous Sensing Systems With Complementary Spatiotemporal Characteristics. Frontiers in Built Environment, p.167. https://doi.org/10.3389/fbuil.2020.560497 · DOI
  20. Marszalek, M., Laptev, I. and Schmid, C., 2009, June. Actions in context. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (pp. 2929–2936). IEEE. https://doi.org/10.1109/CVPR.2009.5206557 · DOI
  21. Yao, B., Khosla, A. and Fei-Fei, L., 2011. Classifying actions and measuring action similarity by modeling the mutual context of objects and human poses. a) A, 1(D2), p.D3. https://www.semanticscholar.org/paper/Classifying-Actions-and-Measuring-Action-Similarity-Yao-Khosla/9612fd66fcd3902bc267a62c146398eb8d30830e
  22. Kapidis, G., Ronald P., Elsbeth V. D., Lucas Noldus, and Remco Veltkamp. "Egocentric hand track and object-based human action recognition." In 2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), pp. 922–929. IEEE, 2019. https://doi.org/10.1109/SmartWorld-UIC-ATC-SCALCOM-IOP-SCI.2019.00185 · DOI
  23. Li, Y., Ji, B., Shi, X., Zhang, J., Kang, B. and Wang, L., 2020. Tea: Temporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 909–918). https://doi.org/10.48550/arXiv.2004.01398 · DOI
  24. Chan, J.Y.L., Bea, K.T., Leow, S.M.H., Phoong, S.W. and Cheng, W.K., 2023. State of the art: a review of sentiment analysis based on sequential transfer learning. Artificial Intelligence Review, 56(1), pp.749–780. https://doi.org/10.1007/s10462-022-10183-8 · DOI
  25. Kumar, Y. and Gupta, S., 2023. Deep transfer learning approaches to predict glaucoma, cataract, choroidal neovascularization, diabetic macular edema, drusen and healthy eyes: an experimental review. Archives of Computational Methods in Engineering, 30(1), pp.521–541. https://doi.org/10.1007/s11831-022-09807-7 · DOI
  26. Fu, Z., He, X., Wang, E., Huo, J., Huang, J. and Wu, D., 2021. Personalized human activity recognition based on integrated wearable sensor and transfer learning. Sensors, 21(3), p.885. https://doi.org/10.3390/s21030885 · DOI · Paper page
  27. Akhand, M.A.H., Roy, S., Siddique, N., Kamal, M.A.S. and Shimamura, T., 2021. Facial Emotion Recognition Using Transfer Learning in the Deep CNN. Electronics, 10(9), p.1036. https://doi.org/10.3390/electronics10091036 · DOI
  28. Cho, J., Baskar, M.K., Li, R., Wiesner, M., Mallidi, S.H., Yalta, N., Karafiat, M., Watanabe, S. and Hori, T., 2018, December. Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling. In 2018 IEEE Spoken Language Technology Workshop (SLT) (pp. 521–527). IEEE. https://doi.org/10.1109/SLT.2018.8639655 · DOI
  29. Khan, M.A., Akram, T., Zhang, Y.D. and Sharif, M., 2021. Attributes based skin lesion detection and recognition: A mask RCNN and transfer learning-based deep learning framework. Pattern Recognition Letters, 143, pp.58–66. https://doi.org/10.1016/j.patrec.2020.12.015 · DOI
  30. Mekruksavanich, S. and Jitpattanakul, A., 2022. Multimodal wearable sensing for sport-related activity recognition using deep learning networks. Journal of Advances in Information Technology. https://doi.org/10.12720/jait.13.2.132-138 · DOI
  31. Zhang, C., Zou, Y., Chen, G. and Gan, L., 2020. Pan: Towards fast action recognition via learning persistence of appearance. arXiv preprint arXiv:2008.03462. https://arxiv.org/abs/2008.03462 · arXiv
  32. Kobayashi, T., Aoki, Y., Shimizu, S., Kusano, K. and Okumura, S., 2019, November. Fine-grained action recognition in assembly work scenes by drawing attention to the hands. In 2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) (pp. 440–446). IEEE. https://ieeexplore.ieee.org/document/9067967
  33. Jones, J.D., Cortesa, C., Shelton, A., Landau, B., Khudanpur, S. and Hager, G.D., 2021. Fine-grained activity recognition for assembly videos. IEEE Robotics and Automation Letters, 6(2), pp.3728–3735. https://doi.org/10.1109/LRA.2021.3064149 · DOI
  34. Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.L. and Grundmann, M., 2020. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214. https://doi.org/10.48550/arXiv.2006.10214 · DOI
  35. Chen, H., Tao, W., Leu, M.C. and Yin, Z., 2020, July. Dynamic gesture design and recognition for human-robot collaboration with convolutional neural networks. In International Symposium on Flexible Automation (Vol. 83617, p. V001T09A001). American Society of Mechanical Engineers. https://doi.org/10.1115/ISFA2020-9609 · DOI
  36. He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). https://doi.org/10.48550/arXiv.1512.03385 · DOI · Paper page
  37. Simonyan, K. and Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. https://doi.org/10.48550/arXiv.1409.1556 · DOI · Paper page
  38. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z., 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818–2826). https://doi.org/10.48550/arXiv.1512.00567 · DOI · Paper page
  39. Tian, C., Xu, Y. and Zuo, W., 2020. Image denoising using deep CNN with batch renormalization. Neural Networks, 121, pp.461–473. https://doi.org/10.1016/j.neunet.2019.08.022 · DOI
  40. Yu, Y., Si, X., Hu, C. and Zhang, J., 2019. A review of recurrent neural networks: LSTM cells and network architectures. Neural computation, 31(7), pp.1235–1270. https://doi.org/10.1162/neco_a_01199 · DOI
  41. Cho, K., Van Merriënboer, B., Bahdanau, D. and Bengio, Y., 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259. https://doi.org/10.48550/arXiv.1409.1259 · DOI
  42. Soomro, K., Zamir, A.R. and Shah, M., 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. https://doi.org/10.48550/arXiv.1212.0402 · DOI · Paper page
  43. Carreira, J. and Zisserman, A., 2017. Quo vadis, action recognition, a new model, and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6299–6308). https://doi.org/10.1109/CVPR.2017.502 · DOI · Paper page
  44. Crasto, N., Weinzaepfel, P., Alahari, K. and Schmid, C., 2019. Mars: Motion-augmented RGB stream for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7882–7891). https://doi.org/10.1109/CVPR.2019.00807 · DOI
  45. Zhu, L., Tran, D., Sevilla-Lara, L., Yang, Y., Feiszli, M. and Wang, H., 2020, April. Faster recurrent networks for efficient video classification. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 07, pp. 13098–13105). https://doi.org/10.1609/aaai.v34i07.7012 · DOI
  46. Qiu, Z., Yao, T., Ngo, C.W., Tian, X. and Mei, T., 2019. Learning Spatio-temporal representation with local and global diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12056–12065). https://doi.org/10.48550/arXiv.1906.05571 · DOI
  47. Stroud, J., Ross, D., Sun, C., Deng, J. and Sukthankar, R., 2020. D3d: Distilled 3d networks for video action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 625–634). https://doi.org/10.48550/arXiv.1812.08249 · DOI
  48. Huang, G. and Bors, A.G., 2022. Busy-Quiet Video Disentangling for Video Classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1341–1350). https://doi.org/10.48550/arXiv.2103.15584 · DOI
  49. Simonyan, K. and Zisserman, A., 2014. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27. https://dl.acm.org/doi/10.5555/2968826.2968890 · DOI
  50. Ma, C.Y., Chen, M.H., Kira, Z. and AlRegib, G., 2021. TS-LSTM and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition. Signal Processing: Image Communication, 71, pp.76–87. https://doi.org/10.1016/j.image.2018.09.003 · DOI
  51. Ullah, B., Kamran, M. and Rui, Y., 2022. Predictive modeling of short-term rockburst for the stability of subsurface structures using machine learning approaches: T-SNE, K-Means clustering and XGBoost. Mathematics, 10(3), p.449. https://doi.org/10.3390/math10030449 · DOI

Metadata

Authors
Chen, H., Zendehdel, N., Leu, M. C., & Yin, Z.
Affiliations
  • Missouri University of Science and Technology (US)
  • Stony Brook University (US)
Corresponding author
Chen, Haodong
ORCID
Venue
Journal of Intelligent Manufacturing
Publisher
Springer Science and Business Media LLC
Published
2023-05-22
Volume
35
Issue
5
Pages
2215-2233
ISSN
0956-5515 (ISSN-L)
DOI
10.21203/rs.3.rs-2387286/v2
OpenAlex
W4380052665
Cited by
47
References
1
Citations by year
2026: 13 · 2025: 30 · 2024: 4
FWCI
4.27
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Intelligent Manufacturing (publishedVersion)
  • Research Square (acceptedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
National Science Foundation (CMMI-1646162, CMMI-1954548)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
Keywords
temporal feature extraction, recurrent neural networks, LSTM, GRU, activity prediction, two-stage network, smart factory, human activity recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review