LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Large-scale datasets have successively proven their fundamental importance in several research fields, especially for early progress in some emerging topics. In this paper, we focus on the problem of visual speech recognition, also known as lip-reading, which has received increasing interest in recent years. We present a naturally-distributed large-scale benchmark for lip-reading in the wild, named LRW-1000, which contains 1,000 classes with 718,018 samples from more than 2,000 individual speakers. Each class corresponds to the syllables of a Mandarin word composed of one or several Chinese characters. To the best of our knowledge, it is currently the largest word-level lipreading dataset and also the only public large-scale Mandarin lip-reading dataset. This dataset aims at covering a "natural" variability over different speech modes and imaging conditions to incorporate challenges encountered in practical applications. It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up. Besides providing a detailed description of the dataset and its collection pipeline, we evaluate several typical popular lip-reading methods and perform a thorough analysis of the results from several aspects. The results demonstrate the consistency and challenges of our dataset, which may open up some new promising directions for future work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Out of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689 · cited 1×
“To tackle this problem, we introduce the SyncNet model in [6], which extracts visual features from 5 frames of cropped faces using a 3D VGG-M network and computes their distance to the MFCC-based audio features.”
MethodChung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · cited 4×
“2% 2015 LRW [5] 500 > 1000 Fixed 256× 256 Natural TV Color 83.”
BackgroundInfluentialCombining Residual Networks with LSTMs for Lipreading
Stafylakis, T., & Tzimiropoulos, G. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-85 · Cited by 348 · cited 2×
“In [21], Stafylakis and Tzimiropoulos proposed to combine two spatial-temporal convolutional layers with a 2D residual network to produce the final representation of the sequence.”
Background
Cited by
Lipreading Using Temporal Convolutional Networks
Martinez, B., Ma, P., Petridis, S., & Pantic, M. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053841 · Cited by 276
Deep learning for visual speech analysis: A survey
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3376710 · Cited by 64
“Meanwhile,theemergence oflarge-scaleVSAdatasets[28],[29],[30],[31],[32]promoted thefurtherdevelopmentofdeeplearningbasedVSAresearch.”
BackgroundMethodCross-Modal Mutual Learning for Audio-Visual Speech Recognition and Manipulation
Yang, C.-C., Fan, W.-C., Yang, C.-F., & Wang, Y.-C. F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20210 · Cited by 16
Metadata
- Authors
- Yang, S., Zhang, Y., Feng, D., Yang, M., Wang, C., Xiao, J., Long, K., Shan, S., & Chen, X.
- Venue
- 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 1-8
- ISSN
- 1541-5058 (ISSN-L)
- OpenAlex
- W2897492880
- Cited by
- 174
- References
- 5
- Type
- conference abstract
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.