The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Lipreading With Local Spatiotemporal Descriptors
Abstract
Visual speech information plays an important role in lipreading under noisy conditions or for listeners with a hearing impairment. In this paper, we present local spatiotemporal descriptors to represent and recognize spoken isolated phrases based solely on visual input. Spatiotemporal local binary patterns extracted from mouth regions are used for describing isolated phrase sequences. In our experiments with 817 sequences from ten phrases and 20 speakers, promising accuracies of 62% and 70% were obtained in speaker-independent and speaker-dependent recognition, respectively. In comparison with other methods on AVLetters database, the accuracy, 62.8%, of our method clearly outperforms the others. Analysis of the confusion matrix for 26 English letters shows the good clustering characteristics of visemes for the proposed descriptors. The advantages of our approach include local processing and robustness to monotonic gray-scale changes. Moreover, no error prone segmentation of moving lips is needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Face Description with Local Binary Patterns: Application to Face Recognition
Ahonen, T., Hadid, A., & Pietikainen, M. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.244 · Cited by 5,584
Self-citationDynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Self-citationXM2VTSDB: The Extended M2VTS Database
Messer, K., Matas, J., Kittler, J., Luettin, J., & Maître, G. (1999) · Cited by 1,158
Recent advances in the automatic recognition of audiovisual speech
Pomianos, G., Neti, C., Gravier, G., Garg, A., & Senior, A. W. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817150 · Cited by 752
Extraction of visual features for lipreading
Matthews, I., Cootes, T. F., Bangham, J. A., Cox, S., & Harvey, R. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982900 · Cited by 537
Modeling Coarticulation in Synthetic Visual Speech
Cohen, M. M., & Massaro, D. W. (1993) · Models and Techniques in Computer Animation · doi:10.1007/978-4-431-66911-1_13 · Cited by 445
BiolD: a multimodal biometric identification system
Frischholz, R. W., & Dieckmann, U. (2000) · Computer · doi:10.1109/2.820041 · Cited by 346
Dynamic Bayesian Networks for Audio-Visual Speech Recognition
Nefian, A. V., Liang, L., Pi, X., Liu, X., & Murphy, K. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206083 · Cited by 320
Audio-visual speech recognition
Neti, C., Potamianos, G., Luettin, J., Matthews, I., Glotin, H., Vergyri, D., Sison, J. A. G., & Mashari, A. (2000) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 319
Boosting Local Binary Pattern (LBP)-Based Face Recognition
Zhang, G., Huang, X., Li, S. Z., Wang, Y., & Wu, X. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30548-4_21 · Cited by 271
Audio-visual integration in multimodal communication
Chen, T., & Rao, R. R. (1998) · Proceedings of the IEEE · doi:10.1109/5.664274 · Cited by 251
AVICAR: audio-visual speech corpus in a car environment
Lee, B., Hasegawa-Johnson, M., Goudeseune, C., Kamdar, S., Borys, S., Liu, M., & Huang, T. (2004) · Interspeech 2004 · doi:10.21437/interspeech.2004-424 · Cited by 181
An image transform approach for HMM based automatic lipreading
Potamianos, G., Graf, H. P., & Cosatto, E. (2002) · Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269) · doi:10.1109/icip.1998.999008 · Cited by 180
Automatic lipreading by optical‐flow analysis
Mase, K., & Pentland, A. (1991) · Systems and Computers in Japan · doi:10.1002/scj.4690220607 · Cited by 165
Chiou, G. I., & Hwang, J. (1997) · IEEE Transactions on Image Processing · doi:10.1109/83.605417 · Cited by 142
2D Cascaded AdaBoost for Eye Localization
Niu, Z., Shan, S., Yan, S., Chen, X., & Gao, W. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.1194 · Cited by 135
Boosted multi-resolution spatiotemporal descriptors for facial expression recognition
Zhao, G., & Pietikäinen, M. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2009.03.018 · Cited by 132
Self-citationAleksic, P. S., & Katsaggelos, A. K. (2006) · Proceedings of the IEEE · doi:10.1109/jproc.2006.886017 · Cited by 128
Models and Techniques in Computer Animation
Magnenat‐Thalmann, N., & Thalmann, D. (1993) · Springer eBooks · doi:10.1007/978-4-431-66911-1 · Cited by 123
A segment-based audio-visual speech recognizer
Hazen, T. J., Saenko, K., La, C.-H., et al. (2004) · International Conference on Multimodal Interfaces · doi:10.1145/1027933.1027972 · Cited by 108
Audio-Visual Speech Recognition Using MPEG-4 Compliant Visual Features
Aleksic, P. S., Williams, J. J., Wu, Z., & Katsaggelos, A. K. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206162 · Cited by 58
Sanderson, C. (2002) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 56
Learning Personal Specific Facial Dynamics for Face Recognition from Videos
Hadid, A., Pietikäinen, M., & Li, S. Z. (2007) · Springer eBooks · doi:10.1007/978-3-540-75690-3_1 · Cited by 48
Audio-to-Visual Conversion Using Hidden Markov Models
Lee, S., & Yook, D. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45683-x_60 · Cited by 48
Person identification using automatic integration of speech, lip, and face experts
Fox, N. A., Ralph Gross, A., de Chazal, P., Cohn, J. F., & Reilly, R. B. (2003) · doi:10.1145/982507.982512 · Cited by 38
Confusability of Phonemes Grouped According to their Viseme Classes in Noisy Environments
Lucey, P., Martin, T. Z., & Sridharan, S. (2004) · QUT ePrints (Queensland University of Technology) · Cited by 35
Audio-Visual Speech Recognition With A Hybrid Svm-Hmm System
Gurban, M., & Thiran, J. (2005) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.5281/zenodo.38970 · Cited by 33
Production Domain Modeling of Pronunciation for Visual Speech Recognition
Saenko, K., Livescu, K., Glass, J. R., & Darrell, T. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1416343 · Cited by 25
Principal appearance and motion from boosted spatiotemporal descriptors
Zhao, G., & Pietikäinen, M. (2008) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4563174 · Cited by 9
Self-citation
Cited by
(2011) · Cited by 2,752
Facial expression recognition from near-infrared videos
Zhao, G., Huang, X., Taini, M., Li, S. Z., & Pietikäinen, M. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.002 · Cited by 778
Self-citationLocal Tetra Patterns: A New Feature Descriptor for Content-Based Image Retrieval
Murala, S., Maheshwari, R. P., & Balasubramanian, R. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2188809 · Cited by 679
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537
Rotation-Invariant Image and Video Description With Local Binary Pattern Features
Zhao, G., Ahonen, T., Matas, J., & Pietikainen, M. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2175739 · Cited by 351
Self-citation
Metadata
- Authors
- Zhao, G., Barnard, M., & Pietikainen, M.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2009-11
- Volume
- 11
- Issue
- 7
- Pages
- 1254-1265
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2136155248
- Cited by
- 309
- References
- 32
- FWCI
- 14.97
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.