The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled
Abstract
This work presents a new approach to learning a framebased classifier on weakly labelled sequence data by embedding a CNN within an iterative EM algorithm. This allows the CNN to be trained on a vast number of example images when only loose sequence level information is available for the source videos. Although we demonstrate this in the context of hand shape recognition, the approach has wider application to any video recognition task where frame level labelling is not available. The iterative EM algorithm leverages the discriminative ability of the CNN to iteratively refine the frame level annotation and subsequent training of the CNN. By embedding the classifier within an EM framework the CNN can easily be trained on 1 million hand images. We demonstrate that the final classifier generalises over both individuals and data sets. The algorithm is evaluated on over 3000 manually labelled hand shape images of 60 different classes which will be released to the community. Furthermore, we demonstrate its use in continuous sign language recognition on two publicly available large sign language data sets, where it outperforms the current state-of-the-art by a large margin. To our knowledge no previous work has explored expectation maximization without Gaussian mixture models to exploit weak sequence labels for sign language recognition.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
Semi-Supervised Learning Literature Survey
Zhu, X. (2005) · Minds at UW (University of Wisconsin) · Cited by 3,844
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 3,752
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Tompson, J., Stein, M., Lecun, Y., & Perlin, K. (2014) · ACM Transactions on Graphics · doi:10.1145/2629500 · Cited by 801
Koller, O., Forster, J., & Ney, H. (2015) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.09.013 · Cited by 533
Self-citationSimultaneous image classification and annotation
Chong, W., Blei, D., & Li, F.-F. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206800 · Cited by 510
Spelling it out: Real-time ASL fingerspelling recognition
Pugeault, N., & Bowden, R. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130290 · Cited by 415
Self-citationEstimating 3D hand pose from a cluttered image
Athitsos, V., & Sclaroff, S. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211500 · Cited by 379
Hands Deep in Deep Learning for Hand Pose Estimation
Oberweger, M., Wohlhart, P., & Lepetit, V. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.06807 · Cited by 298
Real-Time Articulated Hand Pose Estimation Using Semi-supervised Transductive Regression Forests
Tang, D., Yu, T.-H., & Kim, T.-K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.400 · Cited by 219
Learning sign language by watching TV (using weakly aligned subtitles)
Buehler, P., Zisserman, A., & Everingham, M. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206523 · Cited by 173
Efficient Mining of Frequent and Distinctive Feature Configurations
Quack, T., Ferrari, V., Leibe, B., & Van Gool, L. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408906 · Cited by 167
A Linguistic Feature Vector for the Visual Interpretation of Sign Language
Bowden, R., Windridge, D., Kadir, T., Zisserman, A., & Brady, M. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24670-1_30 · Cited by 149
Self-citationThe significance of facial features for automatic sign language recognition
von Agris, U., Knorr, M., & Kraiss, K. (2008) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2008.4813472 · Cited by 148
Extensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather
Forster, J., Schmidt, C., Koller, O., Bellgardt, M., & Ney, H. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/27pdd6xpkah6 · Cited by 140
Self-citationYang, Y., Saleemi, I., & Shah, M. A. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.253 · Cited by 120
Learning signs from subtitles: A weakly supervised approach to sign language recognition
Cooper, H., & Bowden, R. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206647 · Cited by 78
Self-citationRASR - The RWTH Aachen University Open Source Speech Recognition Toolkit
Rybach, D., Hahn, S., Lehnen, P., Nolden, D., Sundermeyer, M., Tüske, Z., Wiesler, S., Schlüter, R., & Ney, H. (2011) · RWTH Publications (RWTH Aachen) · Cited by 77
Self-citationRule of thumb: Deep derotation for improved fingertip detection
Wetzler, A., Slossberg, R., & Kimmel, R. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.33 · Cited by 64
Tracking Using Dynamic Programming for Appearance-Based Sign Language Recognition
Dreuw, P., Thomas Deselaers, B., Rybach, D., Keysers, D., & Ney, H. (2006) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fgr.2006.107 · Cited by 60
Self-citationExtraction of 3D hand shape and posture from image sequences for sign language recognition
Fillbrandt, H., Akyol, S., & Kraiss, K. (2004) · Proceedings - IEEE International SOI Conference/Proceedings · doi:10.1109/amfg.2003.1240841 · Cited by 53
Enhanced continuous sign language recognition using PCA and neural network features
Gweth, Y., Plahl, C., & Ney, H. (2012) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2012.6239187 · Cited by 41
Self-citationModality Combination Techniques for Continuous Sign Language Recognition
Forster, J., Oberdörfer, C., Koller, O., & Ney, H. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38628-2_10 · Cited by 37
Self-citationAligning ASL for Statistical Translation Using a Discriminative Word Model
Farhadi, A., & Forsyth, D. A. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.51 · Cited by 34
Self-supervised learning for object recognition based on kernel discriminant-EM algorithm
Wu, Y. N., Huang, T. S., & Toyama, K. (2002) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2001.937529 · Cited by 33
Yamashita, T., & Watasue, T. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025171 · Cited by 31
Nearest neighbor search methods for handshape recognition
Potamias, M., & Athitsos, V. (2008) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/1389586.1389622 · Cited by 27
May the force be with you: Force-aligned signwriting for automatic subunit annotation of corpora
Koller, O., Ney, H., & Bowden, R. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553777 · Cited by 22
Self-citationRead My Lips: Continuous Signer Independent Weakly Supervised Viseme Recognition
Koller, O., Ney, H., & Bowden, R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_19 · Cited by 21
Self-citation
Cited by
A Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017
Neural Sign Language Translation
Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., & Bowden, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00812 · Cited by 784
Self-citationVideo-Based Sign Language Recognition Without Temporal Segmentation
Huang, J., Zhou, W., Zhang, Q., Li, H., & Li, W. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11903 · Cited by 451
Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective
Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., Caselli, N., Huenerfauth, M., Kacorri, H., Verhoef, T., et al. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353774 · Cited by 422
Self-citationA Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training
Cui, R., Liu, H., & Zhang, C. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2889563 · Cited by 421
Metadata
- Authors
- Koller, O., Ney, H., & Bowden, R.
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 3793-3802
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2463640844
- Cited by
- 278
- References
- 32
- FWCI
- 18.01
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.