The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Improving Sign Language Translation with Monolingual Data by Sign Back-Translation
Open accessAbstract
Despite existing pioneering works on sign language translation (SLT), there is a non-trivial obstacle, i.e., the limited quantity of parallel sign-text data. To tackle this parallel data bottleneck, we propose a sign back-translation (SignBT) approach, which incorporates massive spoken language texts into SLT training. With a text-to-gloss translation model, we first back-translate the monolingual text to its gloss sequence. Then, the paired sign sequence is generated by splicing pieces from an estimated gloss-to-sign bank at the feature level. Finally, the synthetic parallel data serves as a strong supplement for the end-to-end training of the encoder-decoder SLT framework.To promote the SLT research, we further contribute CSL-Daily, a large-scale continuous SLT dataset. It provides both spoken language translations and gloss-level annotations. The topic revolves around people’s daily lives (e.g., travel, shopping, medical care), the most likely SLT application scenario. Extensive experimental results and analysis of SLT methods are reported on CSL-Daily. With the proposed sign back-translation method, we obtain a substantial improvement over previous state-of-the-art SLT methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
(2004) · Cited by 9,718
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 2×
“The encoder-decoder based methods [2, 28, 42] are adopted to process the difference in word order and vocabulary between sign language and spoken language.”
Error bounds for convolutional codes and an asymptotically optimum decoding algorithm
Viterbi, A. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1054010 · Cited by 6,759
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 3×
“The beam search [50] strategy is used to estimate a better decoding path in an acceptable range.”
BackgroundMethodInfluentialBleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,672
Real-time American sign language recognition using desk and wearable computer based video
Starner, T., Weaver, J., & Pentland, A. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.735811 · Cited by 1,335
Unit selection in a concatenative speech synthesis system using a large speech database
Hunt, A. J., & Black, A. W. (2002) · 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings · doi:10.1109/icassp.1996.541110 · Cited by 1,190
When does label smoothing help?
Müller, R., Kornblith, S., & Hinton, G. (2019) · arXiv · doi:10.48550/arxiv.1906.02629 · Cited by 859
Neural Sign Language Translation
Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., & Bowden, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00812 · Cited by 784
Lip reading sentences in the wild
Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.367 · Cited by 702
Li, D., Opazo, C. R., Yu, X., & Li, H. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093512 · Cited by 641
Self-citationAutomatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning
Ong, S. C. W., & Ranganath, S. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.112 · Cited by 562
Koller, O., Forster, J., & Ney, H. (2015) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.09.013 · Cited by 533
Taylor, P. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511816338 · Cited by 529
On Using Monolingual Corpora in Neural Machine Translation
Gülçehre, Ç., Fırat, O., Xu, K., Cho, K., Barrault, L., Lin, H.-C., Bougares, F., Schwenk, H., & Bengio, Y. (2015) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1503.03535 · Cited by 466
Findings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa-jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) · doi:10.18653/v1/w19-5301 · Cited by 461
Video-Based Sign Language Recognition Without Temporal Segmentation
Huang, J., Zhou, W., Zhang, Q., Li, H., & Li, W. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11903 · Cited by 451
Self-citationSign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective
Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., Caselli, N., Huenerfauth, M., Kacorri, H., Verhoef, T., et al. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353774 · Cited by 422 · cited 3×
“Due to the significant social impact and the cross-modality challenge, sign language understanding has been attracting more and more research attention [1, 4, 10, 17, 22, 26, 37, 48].”
A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training
Cui, R., Liu, H., & Zhang, C. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2889563 · Cited by 421
SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition
Camgoz, N. C., Hadfield, S., Koller, O., & Bowden, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.332 · Cited by 365
MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign LanguagePreprint
Joze, H. R. V., & Koller, O. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01053 · Cited by 356
Animating Arbitrary Objects via Deep Motion Transfer
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00248 · Cited by 349
Koller, O., Camgoz, N. C., Ney, H., & Bowden, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2911077 · Cited by 342
Cui, R., Liu, H., & Zhang, C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.175 · Cited by 338
Iterative Alignment Network for Continuous Sign Language Recognition
Pu, J., Zhou, W., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00429 · Cited by 275
Self-citationCLUE: A Chinese Language Understanding Evaluation Benchmark
Xu, L., Hu, H., Zhang, X., Li, L., Cao, C., Li, Y., Xu, Y., Sun, K., Yu, D., Yu, C., et al. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.419 · Cited by 252
Self-citationINCLUDE: A Large Scale Dataset for Indian Sign Language Recognition
Sridhar, A., Ganesan, R. G., Kumar, P., & Khapra, M. M. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413528 · Cited by 226
Multi-channel Transformers for Multi-articulatory Sign Language Translation
Camgöz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66823-5_18 · Cited by 215
TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language TranslationPreprint
Li, D., Xu, C., Yu, X., Zhang, K., Swift, B., Suominen, H., & Li, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05468 · Cited by 207
Self-citationSpatial-Temporal Multi-Cue Network for Sign Language Recognition and Translation
Zhou, H., Zhou, W., Zhou, Y., & Li, H. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3059098 · Cited by 179
Self-citationBSL-1K: Scaling Up Co-articulated Sign Language Recognition Using Mouthing Cues
Albanie, S., Varol, G., Momeni, L., Afouras, T., Chung, J. S., Fox, N., & Zisserman, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_3 · Cited by 176
Progressive Transformers for End-to-End Sign Language Production
Saunders, B., Camgoz, N. C., & Bowden, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_40 · Cited by 171
Chinese sign language recognition with adaptive HMM
Zhang, J., Zhou, W., Xie, C., Pu, J., & Li, H. (2016) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2016.7552950 · Cited by 165
Self-citationTransferring Cross-Domain Knowledge for Video Sign Language Recognition
Li, D., Yu, X., Xu, C., Petersson, L., & Li, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00624 · Cited by 158
Self-citationNiu, Z., & Mak, B. K.-W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_11 · Cited by 137
Boosting Continuous Sign Language Recognition via Cross Modality Augmentation
Pu, J., Zhou, W., Hu, H., & Li, H. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413931 · Cited by 124
Self-citationDynamic Pseudo Label Decoding for Continuous Sign Language Recognition
Zhou, H., Zhou, W., & Li, H. (2019) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00223 · Cited by 122
Self-citationConnectionist Temporal Fusion for Sign Language Translation
Wang, S., Guo, D., Zhou, W., Zha, Z.-J., & Wang, M. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240671 · Cited by 111
Self-citationFingerspelling Recognition in the Wild With Iterative Visual Attention
Shi, B., Rio, A. M. D., Keane, J., Brentari, D., Shakhnarovich, G., & Livescu, K. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00550 · Cited by 107
Isolated Sign Language Recognition with Grassmann Covariance Matrices
Wang, H., Chai, X., Hong, X., Zhao, G., & Chen, X. (2016) · ACM Transactions on Accessible Computing · doi:10.1145/2897735 · Cited by 102
Deep Gesture Video Generation With Learning on Regions of Interest
Cui, R., Cao, Z. L., Pan, W., Zhang, C., & Wang, J. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2960700 · Cited by 33
Cited by
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Metadata
- Authors
- Zhou, H., Zhou, W., Qi, W., Pu, J., & Li, H.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 1316-1325
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3173290664
- Cited by
- 282
- References
- 51
- FWCI
- 24.09
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.