The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Self-supervised representation learning for robust fine-grained human hand action recognition in industrial assembly lines
Open accessPublished version of a preprint
Abstract
Humans are still indispensable on industrial assembly lines, but in the event of an error, they need support from intelligent systems. In addition to the objects to be observed, it is equally important to understand the fine-grained hand movements of a human to be able to track the entire process. However, these deep-learning-based hand action recognition methods are very label intensive, which cannot be offered by all industrial companies due to the associated costs. This work therefore presents a self-supervised learning approach for industrial assembly processes that allows a spatio-temporal transformer architecture to be pre-trained on a variety of information from real-world video footage of daily life. Subsequently, this deep learning model is adapted to the industrial assembly task at hand using only a few labels. Well-known real-world datasets best suited for representation learning of such hand actions in a regression tasks are outlined and to what extent they optimize the subsequent supervised trained classification task. This subsequent fine-tuning is supplemented by concept drift detection, which makes the resulting productively employed models more robust against concept drift and future changing assembly movements.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 3×
“One of the more popular recent work from [10 ] used as autoencoder a Vision Transformer (ViT) [5 ] architecture and found that significantly more masking than the BERT approach’s 15% can be applied to images.”
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
PAGE, E. S. (1954) · Biometrika · doi:10.1093/biomet/41.1-2.100 · Cited by 5,739
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“In the context of this work, however, a curricu - lum learning approach [1 ] is suggested for each batch.”
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735
Learning from Time-Changing Data with Adaptive Windowing
Bifet, A., & Gavaldà, R. (2007) · Proceedings of the 2007 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972771.42 · Cited by 1,789
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471
A Transformer-based Framework for Multivariate Time Series Representation Learning
Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., & Eickhoff, C. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467401 · Cited by 1,278
SimMIM: a Simple Framework for Masked Image Modeling
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00943 · Cited by 1,255
Ensemble learning for data stream analysis: A survey
Krawczyk, B., Minku, L. L., Gama, J., Stefanowski, J., & Woźniak, M. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.004 · Cited by 1,050
The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling
Ho, Y., & Wookey, S. (2020) · IEEE Access · doi:10.1109/access.2019.2962617 · Cited by 835
Webb, G. I., Hyde, R., Cao, H., Nguyen, H. L., & Petitjean, F. (2016) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-015-0448-4 · Cited by 536
Anticipating Visual Representations from Unlabeled Video
Vondrick, C., Pirsiavash, H., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.18 · Cited by 524
DDD: A New Ensemble Approach for Dealing with Concept Drift
Minku, L. L., & Yao, X. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.58 · Cited by 431
An Overview of Concept Drift Applications
Žliobaitė, I., Pechenizkiy, M., & Gama, J. (2016) · Studies in Big Data · doi:10.1007/978-3-319-26989-4_4 · Cited by 379
Tracking Emerges by Colorizing Videos
Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_24 · Cited by 379
Masked Autoencoders as Spatiotemporal Learners
Feichtenhofer, C., Fan, H., Li, Y., & He, K. (2022) · neural information processing systems · doi:10.52202/068431-2605 · Cited by 273
Discussion and review on evolving data streams and concept drift adapting
Khamassi, I., Sayed-Mouchaweh, M., Hammami, M., & Ghédira, K. (2016) · Evolving Systems · doi:10.1007/s12530-016-9168-2 · Cited by 224
Temporal quality degradation in AI models
Jarquin, D. V., Sharp, A., Zhang, R., Nguyen, T., Hoang, A. T., & Pianykh, O. S. (2022) · Scientific Reports · doi:10.1038/s41598-022-15245-z · Cited by 216
An Overview on Concept Drift Learning
Iwashita, A. S., & Papa, J. P. (2018) · IEEE Access · doi:10.1109/access.2018.2886026 · Cited by 179
Data stream mining: methods and challenges for handling concept drift
Wares, S., Isaacs, J. P., & Elyan, E. (2019) · SN Applied Sciences · doi:10.1007/s42452-019-1433-0 · Cited by 127
SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-Supervised Skeleton Action Recognition
Wu, W., Hua, Y., Zheng, C., Wu, S., Chen, C., & Lu, A. (2023) · 2023 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) · doi:10.1109/icmew59549.2023.00045 · Cited by 60 · cited 2×
“[ 44] presented SkeletonMAE, a MAE approach that considers 3D skeletal data.”
Don’t Pay for Validation: Detecting Drifts from Unlabeled data Using Margin Density
Sethi, T. S., & Kantardzic, M. M. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.07.284 · Cited by 53
MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting
Tang, P., & Zhang, X. (2022) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai56018.2022.00150 · Cited by 40
Supporting the Page-Hinkley Test with Empirical Mode Decomposition for Change Detection
Sebastião, R., & Fernandes, J. M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-60438-1_48 · Cited by 37
Sturm, F., Hergenröther, E., Reinhardt, J., Vojnovikj, P. S., & Siegel, M. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-37717-4_70 · Cited by 8
Self-citationSturm, F., Sathiyababu, R., Allipilli, H., Hergenröther, E., & Siegel, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47969-4_14 · Cited by 4
Self-citation
Research works printed but not linked: 14
- Cao, S., Xu, P., Clifton, D.A.: How to understand masked autoencoders. arXiv preprint arXiv:2202.03670 (2022) · arXiv
- Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., Bouchachia, A.: A survey on concept drift adaptation. ACM Comput. Surv. (CSUR) 46(4), 1–37 (2014) · DOI · Paper page
- Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016) · arXiv
- Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Bach, F.R., Blei, D.M. (eds.) Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015. JMLR Workshop and Conference Proceedings, vol. 37, pp. 448–456. JMLR.org (2015), http://proceedings.mlr.press/v37/ioffe15.html
- Li, Y., Liu, M., Rehg, J.M.: In the eye of the beholder: Gaze and actions in first person video (2020). arxiv:2006.00626 · arXiv
- Lin, T., Dollár, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. CoRR abs/1612.03144 (2016), arxiv:1612.03144 · arXiv
- Liu, M., Ren, S., Ma, S., Jiao, J., Chen, Y., Wang, Z., Song, W.: Gated transformer networks for multivariate time series classification. CoRR abs/2103.14438 (2021), arxiv:2103.14438 · arXiv
- Mahdisoltani, F., Berger, G., Gharbieh, W., Fleet, D.J., Memisevic, R.: Fine-grained video classification and captioning. CoRR abs/1804.09235 (2018), arxiv:1804.09235 · arXiv
- Tong, Z., Song, Y., Wang, J., Wang, L.: Videomae: masked autoencoders are data-efficient learners for self-supervised video pre-training. Adv. Neural. Inf. Process. Syst. 35, 10078–10093 (2022)
- Trockman, A., Kolter, J.Z.: Patches are all you need? Trans. Mach. Learn. Res. 2023 (2022)
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)
- Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, ll (Dec) pp. 3371–3408 (2010)
- Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A., Bottou, L.: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11(12), 3371–3408 (2010)
- Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C., Grundmann, M.: Mediapipe hands: On-device real-time hand tracking. CoRR abs/2006.10214 (2020), arxiv:2006.10214 · arXiv
Metadata
- Authors
- Sturm, F., Trat, M., Sathiyababu, R., Allipilli, H., Menz, B., Hergenroether, E., & Siegel, M.
- Affiliations
- Darmstadt University of Applied Sciences (DE)
- FZI Research Center for Information Technology (DE)
- Georg Simon Ohm University of Applied Sciences Nuremberg (DE)
- Corresponding author
- Sturm, Fabian
- Venue
- Machine Vision and Applications
- Published
- 2024-12-13
- Volume
- 36
- Issue
- 1
- ISSN
- 0932-8092 (ISSN-L)
- 1432-1769
- OpenAlex
- W4405385334
- Cited by
- 2
- References
- 30
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 0.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Machine Vision and Applications (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- Funders
- Technische Hochschule Nürnberg
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- self-supervised representation learning, industrial assembly lines, spatio-temporal transformer, few-shot fine-tuning, concept drift detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
