The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos
Abstract
In this work we present a new approach to the field of weakly supervised learning in the video domain. Our method is relevant to sequence learning problems which can be split up into sub-problems that occur in parallel. Here, we experiment with sign language data. The approach exploits sequence constraints within each independent stream and combines them by explicitly imposing synchronisation points to make use of parallelism that all sub-problems share. We do this with multi-stream HMMs while adding intermediate synchronisation constraints among the streams. We embed powerful CNN-LSTM models in each HMM stream following the hybrid approach. This allows the discovery of attributes which on their own lack sufficient discriminative power to be identified. We apply the approach to the domain of sign language recognition exploiting the sequential parallelism to learn sign language, mouth shape and hand shape classifiers. We evaluate the classifiers on three publicly available benchmark data sets featuring challenging real-life sign language with over 1,000 classes, full sentence based lip-reading and articulated hand shape recognition on a fine-grained hand shape taxonomy featuring over 60 different hand shapes. We clearly outperform the state-of-the-art on all data sets and observe significantly faster convergence using the parallel alignment approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,839
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“Comparable to [4] andwith a similar effect as ‘curriculum learning’ [69], we observe fastest convergence if we first train a CNN-HMM model with randomly shuffled input images for 4 EM re-alignment iterations starting from a flat start in each stream.”
BackgroundA Systematic Comparison of Various Statistical Alignment Models
Och, F. J., & Ney, H. (2003) · Computational Linguistics · doi:10.1162/089120103321337421 · Cited by 3,854
Self-citationSemi-Supervised Learning Literature Survey
Zhu, X. (2005) · Minds at UW (University of Wisconsin) · Cited by 3,844
Statistical Inference for Probabilistic Functions of Finite State Markov Chains
Baum, L. E., & Petrie, T. (1966) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177699147 · Cited by 2,775
Factorial Hidden Markov Models
Ghahramani, Z., & Jordan, M. I. (1997) · Machine Learning · doi:10.1023/a:1007425814087 · Cited by 1,157
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
Coupled hidden Markov models for complex action recognition
Brand, M., Oliver, N., & Pentland, A. (2002) · Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1997.609450 · Cited by 1,052
Neural Sign Language Translation
Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., & Bowden, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00812 · Cited by 784
Self-citationRecent advances in the automatic recognition of audiovisual speech
Pomianos, G., Neti, C., Gravier, G., Garg, A., & Senior, A. W. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817150 · Cited by 752
Lip reading sentences in the wild
Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.367 · Cited by 702
Some statistical issues in the comparison of speech recognition algorithms
Gillick, L., & Cox, S. J. (2003) · International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1989.266481 · Cited by 636
Joint-sequence models for grapheme-to-phoneme conversion
Bisani, M., & Ney, H. (2008) · Speech Communication · doi:10.1016/j.specom.2008.01.002 · Cited by 599
Self-citationAudio-visual speech modeling for continuous speech recognition
Dupont, S., & Luettin, J. (2000) · IEEE Transactions on Multimedia · doi:10.1109/6046.865479 · Cited by 598
Koller, O., Forster, J., & Ney, H. (2015) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.09.013 · Cited by 533
Self-citationSimultaneous image classification and annotation
Chong, W., Blei, D., & Li, F.-F. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206800 · Cited by 510
Hidden Markov model decomposition of speech and noise
Varga, A. P., & Moore, R. K. (2002) · International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1990.115970 · Cited by 467
Gradient-based learning algorithms for recurrent networks and their computational complexity
Williams, R. J., & Zipser, D. (1995) · Cited by 424
SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition
Camgoz, N. C., Hadfield, S., Koller, O., & Bowden, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.332 · Cited by 365
Self-citationCui, R., Liu, H., & Zhang, C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.175 · Cited by 338
Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition
Koller, O., Zargaran, S., Ney, H., & Bowden, R. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.136 · Cited by 289
Self-citationKoller, O., Ney, H., & Bowden, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.412 · Cited by 278
Self-citationRe-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs
Koller, O., Zargaran, S., & Ney, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.364 · Cited by 266
Self-citationParallel hidden Markov models for American sign language recognition
Vogler, C., & Metaxas, D. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.791206 · Cited by 258
Factorial Hidden Markov Models.
Ghahramani, Z., & Jordan, M. I. (1996) · doi:10.21236/ada307097 · Cited by 254
A mew ASR approach based on independent processing and recombination of partial frequency bands
Bourlard, H., & Dupont, S. (2002) · Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96 · doi:10.1109/icslp.1996.607145 · Cited by 248
Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs
Koller, O., Zargaran, S., Ney, H., & Bowden, R. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1121-3 · Cited by 203
Self-citationLiwicki, M., Graves, A., Horst Bunke, A., & Schmidhuber, J. (2007) · BORIS (University Library Bern) · Cited by 183
Learning sign language by watching TV (using weakly aligned subtitles)
Buehler, P., Zisserman, A., & Everingham, M. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206523 · Cited by 173
Efficient Mining of Frequent and Distinctive Feature Configurations
Quack, T., Ferrari, V., Leibe, B., & Van Gool, L. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408906 · Cited by 167
RWTH-PHOENIX-Weather: A Large Vocabulary Sign Language Recognition and Translation Corpus
Forster, J., Schmidt, C., Hoyoux, T., Koller, O., Zelle, U., Piater, J., & Ney, H. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4vmmiu4jeew5 · Cited by 148
Self-citationExtensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather
Forster, J., Schmidt, C., Koller, O., Bellgardt, M., & Ney, H. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/27pdd6xpkah6 · Cited by 140
Self-citationRecent developments in visual sign language recognition
von Agris, U., Zieren, J., Canzler, U., Bauer, B., & Kraiss, K. (2007) · Universal Access in the Information Society · doi:10.1007/s10209-007-0104-x · Cited by 133
Deep Learning of Mouth Shapes for Sign Language
Koller, O., Ney, H., & Bowden, R. (2015) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2015.69 · Cited by 133
Self-citationTools for the analysis of benchmark speech recognition tests
Pallett, D. S., Fisher, W. M., & Fiscus, J. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1990.115546 · Cited by 132
Yang, Y., Saleemi, I., & Shah, M. A. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.253 · Cited by 120
Integrating audio and visual information to provide highly robust speech recognition
Tomlinson, M. J., Russell, M. J., & Brooke, N. M. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543247 · Cited by 108
Handshapes and Movements: Multiple-Channel American Sign Language Recognition
Vogler, C., & Metaxas, D. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24598-8_23 · Cited by 85
Learning signs from subtitles: A weakly supervised approach to sign language recognition
Cooper, H., & Bowden, R. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206647 · Cited by 78
Self-citationRASR - The RWTH Aachen University Open Source Speech Recognition Toolkit
Rybach, D., Hahn, S., Lehnen, P., Nolden, D., Sundermeyer, M., Tüske, Z., Wiesler, S., Schlüter, R., & Ney, H. (2011) · RWTH Publications (RWTH Aachen) · Cited by 77
Self-citationAn Asynchronous Hidden Markov Model for Audio-Visual Speech Recognition
Samy Bengio, A. (2002) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 71
Multimodal Gesture Recognition Using Multi-stream Recurrent Neural Network
Nishida, N., & Nakayama, H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-29451-3_54 · Cited by 53
Modality Combination Techniques for Continuous Sign Language Recognition
Forster, J., Oberdörfer, C., Koller, O., & Ney, H. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-38628-2_10 · Cited by 37
Self-citationAligning ASL for Statistical Translation Using a Discriminative Word Model
Farhadi, A., & Forsyth, D. A. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.51 · Cited by 34
Self-supervised learning for object recognition based on kernel discriminant-EM algorithm
Wu, Y. N., Huang, T. S., & Toyama, K. (2002) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2001.937529 · Cited by 33
Handshapes and movements: Multiple-channel American Sign Language recognition
Vogler, C., & Metaxas, D. (2004) · Cited by 32
May the force be with you: Force-aligned signwriting for automatic subunit annotation of corpora
Koller, O., Ney, H., & Bowden, R. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553777 · Cited by 22
Self-citationRead My Lips: Continuous Signer Independent Weakly Supervised Viseme Recognition
Koller, O., Ney, H., & Bowden, R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_19 · Cited by 21
Self-citationEnhancing gloss-based corpora with facial features using active appearance models
Schmidt, C., Koller, O., Ney, H., Hoyoux, T., & Piater, J. (2013) · ORBi (University of Liège) · Cited by 16
Self-citation
Cited by
Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective
Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., Caselli, N., Huenerfauth, M., Kacorri, H., Verhoef, T., et al. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353774 · Cited by 422
Self-citationImproving Sign Language Translation with Monolingual Data by Sign Back-Translation
Zhou, H., Zhou, W., Qi, W., Pu, J., & Li, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00137 · Cited by 282
Metadata
- Authors
- Koller, O., Camgoz, N. C., Ney, H., & Bowden, R.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2020-09-01
- Volume
- 42
- Issue
- 9
- Pages
- 2306-2320
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2941870244
- Cited by
- 342
- References
- 59
- FWCI
- 30.78
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.