The DOI matches its Crossref record; 94 of the 94 references it cites were found in a registry or an index.
Recent advances in the automatic recognition of audiovisual speech
Abstract
Visual speech information from the speaker's mouth region has been successfully shown to improve noise robustness of automatic speech recognizers, thus promising to extend their usability in the human computer interface. In this paper, we review the main components of audiovisual automatic speech recognition (ASR) and present novel contributions in two main areas: first, the visual front-end design, based on a cascade of linear image transforms of an appropriate video region of interest, and subsequently, audiovisual speech integration. On the latter topic, we discuss new work on feature and decision fusion combination, the modeling of audiovisual speech asynchrony, and incorporating modality reliability estimates to the bimodal recognition process. We also briefly touch upon the issue of audiovisual adaptation. We apply our algorithms to three multisubject bimodal databases, ranging from small- to large-vocabulary recognition tasks, recorded in both visually controlled and challenging environments. Our experiments demonstrate that the visual modality improves ASR over all conditions and data considered, though less so for visually challenging environments and large vocabulary tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
A Simplex Method for Function Minimization
Nelder, J. A., & Mead, R. (1965) · The Computer Journal · doi:10.1093/comjnl/7.4.308 · Cited by 28,905
Kass, M., Witkin, A., & Terzopoulos, D. (1988) · International Journal of Computer Vision · doi:10.1007/bf00133570 · Cited by 17,070
Linear Statistical Inference and its Applications: Second Editon
Rao, C. R. (1973) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316436 · Cited by 10,489
Fundamentals of speech recognition
Rabiner, L. R., & Juang, B.-H. F. (1993) · Cited by 7,680
Active Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190
Statistical pattern recognition: a review
Jain, A. K., Duin, P. W., & Mao, J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.824819 · Cited by 6,767
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,290
Neural network-based face detection
Rowley, H. A., Baluja, S., & Kanade, T. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.655647 · Cited by 3,554
Visual Contribution to Speech Intelligibility in Noise
Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · cited 1×
“The visual modality benefit to speech intelligibility in noise has been quantified as far back as in 1954 [6].”
BackgroundDiscrete-Time Processing of Speech Signals
Deller, J. R., Hansen, J. H. L., & Proakis, J. G. (1999) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544402 · Cited by 2,458
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0054760 · Cited by 2,195
Leggetter, C. J., & Woodland, P. C. (1995) · Computer Speech & Language · doi:10.1006/csla.1995.0010 · Cited by 2,192
Methods of combining multiple classifiers and their applications to handwriting recognition
Xu, L., Krzyzak, A., & Suen, C. Y. (1992) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.155943 · Cited by 2,179
Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains
Gauvain, J.-L., & Lee, C.-H. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.279278 · Cited by 2,108
Introduction to Multivariate Analysis
Lee, J. C. (1983) · Technometrics · doi:10.1080/00401706.1983.10487903 · Cited by 1,783
Example-based learning for view-based human face detection
Sung, K.-K., & Poggio, T. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.655648 · Cited by 1,773
Young, S. J., Jansen, J., Odell, J. J., Ollason, D., & Woodland, P. (1995) · Cited by 1,427
Feature extraction from faces using deformable templates
Yuille, A. L., Hallinan, P. W., & Cohen, D. S. (1992) · International Journal of Computer Vision · doi:10.1007/bf00127169 · Cited by 1,369
Coupled hidden Markov models for complex action recognition
Brand, M., Oliver, N., & Pentland, A. (2002) · Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1997.609450 · Cited by 1,052
Jain, A. K., Ross, A. A., & Nandakumar, K. (2011) · Springer eBooks · doi:10.1007/978-0-387-77326-1 · Cited by 969
Maximum mutual information estimation of hidden Markov model parameters for speech recognition
Bahl, L., Brown, P., de Souza, P., & Mercer, R. (2005) · ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1986.1169179 · Cited by 817
Linear Statistical Inference and Its Applications
Johnson, N. L. (1966) · Technometrics · doi:10.1080/00401706.1966.10490390 · Cited by 803
Audio-visual speech modeling for continuous speech recognition
Dupont, S., & Luettin, J. (2000) · IEEE Transactions on Multimedia · doi:10.1109/6046.865479 · Cited by 598
Speech recognition by machines and humans
Lippmann, R. P. (1997) · Speech Communication · doi:10.1016/s0167-6393(97)00021-6 · Cited by 529
Hidden Markov model decomposition of speech and noise
Varga, A. P., & Moore, R. K. (2002) · International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1990.115970 · Cited by 467
Some preliminaries to a comprehensive account of audio-visual speech perception.
Summerfield, A. Q. (1987) · Cited by 421
Quantitative association of vocal-tract and facial behavior
Yehia, H., Rubin, P., & Vatikiotis-Bateson, E. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00048-x · Cited by 420
Automatic lipreading to enhance speech recognition (speech reading)
Petajan, E. D. (1984) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 328
Dynamic Bayesian Networks for Audio-Visual Speech Recognition
Nefian, A. V., Liang, L., Pi, X., Liu, X., & Murphy, K. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206083 · Cited by 320
"Eigenlips" for robust speech recognition
Bregler, C., & Konig, Y. (2002) · Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.1994.389567 · Cited by 312
CUAVE: A new audio-visual database for multimodal human-computer interface research
Patterson, E. K., Gurbuz, S., Tufekci, Z., & Gowdy, J. N. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745028 · Cited by 286
Audio-visual speech recognition
Neti, C., Potamianos, G., Luettin, J., Matthews, I. A., Glotin, H., Vergyri, D., Sison, J. A. G., & Mashari, A. (2000) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 272
Self-citationMaximum likelihood modeling with Gaussian distributions for classification
Gopinath, R. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1998.675351 · Cited by 267
Audio-visual integration in multimodal communication
Chen, T., & Rao, R. R. (1998) · Proceedings of the IEEE · doi:10.1109/5.664274 · Cited by 251
A mew ASR approach based on independent processing and recombination of partial frequency bands
Bourlard, H., & Dupont, S. (2002) · Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96 · doi:10.1109/icslp.1996.607145 · Cited by 248
Chen, T. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911195 · Cited by 230
Speechreading by Humans and Machines: Models, Systems, and Applications
(1996) · NATO ASI Series · doi:10.1007/978-3-662-13015-5 · Cited by 226
A review of speech-based bimodal recognition
Chibelushi, C. C., Deravi, F., & Mason, J. S. D. (2002) · IEEE Transactions on Multimedia · doi:10.1109/6046.985551 · Cited by 222
Speech recognition in adverse environments
Juang, B. H. (1991) · Computer Speech & Language · doi:10.1016/0885-2308(91)90011-e · Cited by 198
An image transform approach for HMM based automatic lipreading
Potamianos, G., Graf, H. P., & Cosatto, E. (2002) · Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269) · doi:10.1109/icip.1998.999008 · Cited by 180
On the Integration of Auditory and Visual Parameters in an HMM-based ASR
Adjoudani, A., & Benoît, C. (1996) · NATO ASI Series · doi:10.1007/978-3-662-13015-5_35 · Cited by 161
Chiou, G. I., & Hwang, J. (1997) · IEEE Transactions on Image Processing · doi:10.1109/83.605417 · Cited by 142
The M2VTS multimodal face database (Release 1.00)
Pigeon, S., & Vandendorpe, L. (1997) · Lecture notes in computer science · doi:10.1007/bfb0016021 · Cited by 141
Efficient cepstral normalization for robust speech recognition
Liu, F., Stern, R. M., Huang, X., & Acero, A. (1993) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/1075671.1075688 · Cited by 135
Audio-visual enhancement of speech in noise
Laurent Girin, B., Schwartz, J., & Feng, G. (2001) · The Journal of the Acoustical Society of America · doi:10.1121/1.1358887 · Cited by 130
Improving connected letter recognition by lipreading
Bregler, C., Hild, H., Manke, S., & Waibel, A. H. (1993) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1993.319179 · Cited by 125
Photo-realistic talking-heads from image samples
Cosatto, E., & Graf, H. P. (2000) · IEEE Transactions on Multimedia · doi:10.1109/6046.865480 · Cited by 125
Speechreading using Probabilistic Models
Luettin, J., & Thacker, N. A. (1997) · Computer Vision and Image Understanding · doi:10.1006/cviu.1996.0570 · Cited by 123
Computer lipreading for improved accuracy in automatic speech recognition
Silsbee, P. L., & Bovik, A. C. (1996) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.536928 · Cited by 121
Jain, A. K., Bolle, R. M., & Pankanti, S. (2006) · Springer eBooks · doi:10.1007/0-306-47044-6_1 · Cited by 117
Noise Adaptive Stream Weighting in Audio-Visual Speech Recognition
Heckmann, M., Berthommier, F., & Kroschel, K. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206150 · Cited by 115
On the Relationship between Face Movements, Tongue Movements, and Speech Acoustics
Jiang, J., Alwan, A. A., Keating, P., et al. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206046 · Cited by 114
Integrating audio and visual information to provide highly robust speech recognition
Tomlinson, M. J., Russell, M. J., & Brooke, N. M. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543247 · Cited by 108
Visionary Speech: Looking Ahead to Practical Speechreading Systems
Hennecke, M. E., Stork, D. G., & Prasad, K. V. (1996) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-13015-5_25 · Cited by 106
Discriminative training of HMM stream exponents for audio-visual speech recognition
Potamianos, G., & Graf, H. P. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1998.679695 · Cited by 106
Joint Audio-Visual Tracking Using Particle Filters
Zotkin, D. N., Ramani Duraiswami, B., & Davis, L. S. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206058 · Cited by 95
Comparing models for audiovisual fusion in a noisy-vowel recognition task
Teissier, P., Robert-Ribes, J., Schwartz, J., & Guérin-Dugué, A. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.799688 · Cited by 95
See me, hear me: integrating automatic speech recognition and lip-reading
Duchnowski, P., Meier, U., & Waibel, A. (1994) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1994-139 · Cited by 93
SPEECH INTELLIGIBILITY DERIVED FROM ASYNCHRONOUS PROCESSING OF AUDITORY-VISUAL INFORMATION
Grant, K. W., & Greenberg, S. (2001) · AVSP · Cited by 90
Adaptive bimodal sensor fusion for automatic speechreading
Meier, U., Hürst, W., & Duchnowski, P. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543250 · Cited by 77
Hierarchical discriminant features for audio-visual LVCSR
Potamianos, G., Luettin, J., & Neti, C. (2001) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2001.940793 · Cited by 76
Self-citationAutomatic Speechreading with Applications to Human-Computer Interfaces
Zhang, X., Broun, C. C., Mersereau, R. M., & Clements, M. A. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206137 · Cited by 76
Speechreading using shape and intensity information
Luettin, J., Thacker, N. A., & Beet, S. W. (1996) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607024 · Cited by 70
LIPS, TEETH, AND THE BENEFITS OF LIPREADING
Summerfield, Q., MacLeod, A., McGrath, M., & Brooke, M. (1989) · Handbook of Research on Face Processing · doi:10.1016/b978-0-444-87143-5.50019-6 · Cited by 64
Real-time lip tracking and bimodal continuous speech recognition
Chan, M. T., Zhang, Y., & Huang, T. S. (2002) · IEEE Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.1998.738914 · Cited by 60
Audio-Visual Speech Recognition Using MPEG-4 Compliant Visual Features
Aleksic, P. S., Williams, J. J., Wu, Z., & Katsaggelos, A. K. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206162 · Cited by 58
Adaptive fusion of acoustic and visual sources for automatic speech recognition
Rogozan, A., & Deléglise, P. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00056-9 · Cited by 56
Multi-source face tracking with audio and visual data
Wang, C., & Brandstein, M. S. (1999) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.1999.793815 · Cited by 49
Lip synchronization using speech-assisted video processing
Chen, T., Graf, H. P., & Wang, K. (1995) · IEEE Signal Processing Letters · doi:10.1109/97.376913 · Cited by 49
Sodoyer, D., Schwartz, J., Laurent Girin, B., Klinkisch, J., & Jutten, C. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702207015 · Cited by 46
Maximum entropy and MCE based HMM stream weight estimation for audio-visual ASR
Gravier, G., Axelrod, S., Potamianos, G., & Neti, C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5743873 · Cited by 44
Self-citationA Cascade Visual Front End for Speaker Independent Automatic Speechreading
Potamianos, G., Neti, C., Iyengar, G., Senior, A., & Verma, A. (2001) · International Journal of Speech Technology · doi:10.1023/a:1011352422845 · Cited by 43
Self-citationFeature analysis for automatic speechreading
Scanlon, P., & Reilly, R. B. (2002) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2001.962802 · Cited by 41
Signal Processing for Robust Speech Recognition
Stern, R. M., Acero, A., Liu, F., & Ohshima, Y. (1996) · Kluwer international series in engineering and computer science · doi:10.1007/978-1-4613-1367-0_15 · Cited by 40
Rationale for Phoneme-Viseme Mapping and Feature Selection in Visual Speech Recognition
Goldschen, A. J., Garcia, O. N., & Petajan, E. D. (1996) · NATO ASI series. Series F : Computer and system sciences · doi:10.1007/978-3-662-13015-5_39 · Cited by 39
Noisy audio feature enhancement using audio-visual speech data
Goecke, R., Potamianos, G., & Neti, C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745030 · Cited by 38
Self-citationDynamic Features for Visual Speechreading: A Systematic Comparison
Gray, M. S., Movellan, J. R., & Sejnowski, T. J. (1996) · neural information processing systems · Cited by 37
Improved ROI and within frame discriminant features for lipreading
Potamianos, G., & Neti, C. (2002) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2001.958098 · Cited by 34
Self-citationA MINIMUM ERROR RATE PATTERN RECOGNITION APPROACH TO SPEECH RECOGNITION
Chou, W., LEE, C.-H., Juang, B.-H., & Soong, F. K. (1994) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001494000024 · Cited by 34
Audio-visual speaker recognition for video broadcast news: some fusion techniques
Maison, B., Neti, C., & Senior, A. (1999) · IEEE Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.1999.793814 · Cited by 34
Self-citationAudio-visual unit selection for the synthesis of photo-realistic talking-heads
Cosatto, E., Potamianos, G., & Graf, H. P. (2002) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2000.871439 · Cited by 34
An HMM-based speech-to-video synthesizer
Williams, J. J., & Katsaggelos, A. K. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1021891 · Cited by 32
Bimodal speech recognition using coupled hidden Markov models
Chu, S. M., & Huang, T. S. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-377 · Cited by 31
Audio-visual speech modeling using coupled hidden Markov models
Chu, S. M., & Huang, T. S. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745026 · Cited by 31
AUTOMATIC SPEECHREADING OF IMPAIRED SPEECH
Potamianos, G., & Neti, C. (2001) · Cited by 29
Self-citationFusion of Audio-Visual Information for Integrated Speech Processing
Nakamura, S. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45344-x_20 · Cited by 22
A hybrid ANN/HMM audio-visual speech recognition system.
Heckmann, M., Berthommier, F., & Kroschel, K. (2001) · Cited by 21
What makes a good speechreader? First you have to find one
Leybaert, J., Alegria, J., Hage, C., & Charlier, B. (2013) · Routledge eBooks · doi:10.4324/9780203098752-21 · Cited by 21
A recombination strategy for multi-band speech recognition based on mutual information criterion
Okawa, S., Nakajima, T., & Shirai, K. (1999) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1999-155 · Cited by 17
Mouth movement and signed communication
(2013) · Routledge eBooks · doi:10.4324/9780203098752-24 · Cited by 13
What can visual speech synthesis tell visual speech recognition?
Cohen, M. M., & Massaro, D. W. (2002) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/acssc.1994.471516 · Cited by 10
ADAPTIVE BIMODAL SENSOR FUSION FOR AUTOI/IATIC SPEECHREADING
Meiel, U. (2004) · Cited by 5
Cited by
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
(2011) · Springer eBooks · doi:10.1007/978-0-85729-932-1 · Cited by 1,819
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
An audio-visual corpus for speech perception and automatic speech recognition
Cooke, M., Barker, J., Cunningham, S., & Shao, X. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2229005 · Cited by 1,144
Trusted Multi-View Classification With Dynamic Evidential Fusion
Han, Z., Zhang, C., Fu, H., & Zhou, J. T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3171983 · Cited by 536
Metadata
- Authors
- Pomianos, G., Neti, C., Gravier, G., Garg, A., & Senior, A. W.
- Venue
- Proceedings of the IEEE
- Published
- 2003-09
- Volume
- 91
- Issue
- 9
- Pages
- 1306-1326
- ISSN
- 0018-9219 (ISSN-L)
- OpenAlex
- W2096391593
- Cited by
- 752
- References
- 94
- FWCI
- 38.74
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.