Integration of acoustic and visual speech signals using neural networks

Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations

Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686 · see citing works

Hearing lips and seeing voices

MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · see citing works

“Neural” computation of decisions in optimization problems

Hopfield, J. J., & Tank, D. W. (1985) · Biological Cybernetics · doi:10.1007/bf00339943 · Cited by 6,035 · see citing works

A learning algorithm for Boltzmann machines

Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402 · see citing works

Learning internal representations by error propagation

RUMELHART, D. E., HINTON, G. E., & WILLIAMS, R. J. (1988) · Readings in Cognitive Science · doi:10.1016/b978-1-4832-1446-7.50035-2 · Cited by 3,002 · see citing works

Acoustic Theory of Speech Production

Fant, G. (1971) · De Gruyter eBooks · doi:10.1515/9783110873429 · Cited by 2,958 · see citing works

Control Methods Used in a Study of the Vowels

Peterson, G. E., & Barney, H. L. (1952) · The Journal of the Acoustical Society of America · doi:10.1121/1.1906875 · Cited by 2,837 · see citing works

Digital processing of speech signals

(1980) · Pattern Recognition · doi:10.1016/0031-3203(80)90010-2 · Cited by 2,642 · see citing works

Visual Contribution to Speech Intelligibility in Noise

Sumby, W. H., & Pollack, I. (1954) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907309 · Cited by 2,583 · see citing works

Digital Processing of Speech Signals

Ackroyd, M. H. (1979) · Electronics and Power · doi:10.1049/ep.1979.0173 · Cited by 2,408 · see citing works

Linear Prediction of Speech

Markel, J. D., & Gray, A. H. (1976) · Communication and cybernetics · doi:10.1007/978-3-642-66286-7 · Cited by 2,258 · see citing works

An Analysis of Perceptual Confusions Among Some English Consonants

Miller, G. A., & Nicely, P. E. (1955) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907526 · Cited by 1,885 · see citing works

Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms

White, B. W., & Rosenblatt, F. (1963) · The American Journal of Psychology · doi:10.2307/1419730 · Cited by 1,818 · see citing works

Analog VLSI Implementation of Neural Systems

(1989) · Kluwer international series in engineering and computer science · doi:10.1007/978-1-4613-1639-8 · Cited by 1,684 · see citing works

Digital signal processing

Mark, J. W. (1975) · Proceedings of the IEEE · doi:10.1109/proc.1975.9993 · Cited by 1,569 · see citing works

Parallel Networks that Learn to Pronounce English Text

B, T. J. S. A. (1987) · Cited by 1,524 · see citing works

Speech Analysis Synthesis and Perception

Flanagan, J. L. (1972) · doi:10.1007/978-3-662-01562-9 · Cited by 1,469 · see citing works

Cooperative Computation of Stereo Disparity

Marr, D., & Poggio, T. (1976) · Science · doi:10.1126/science.968482 · Cited by 1,387 · see citing works

Factors Governing the Intelligibility of Speech Sounds

French, N. R., & Steinberg, J. C. (1947) · The Journal of the Acoustical Society of America · doi:10.1121/1.1916407 · Cited by 1,137 · see citing works

Generalization of back-propagation to recurrent neural networks

Pineda, F. J. (1987) · Physical Review Letters · doi:10.1103/physrevlett.59.2229 · Cited by 931 · see citing works

The bimodal perception of speech in infancy

Kuhl, P. K., & Meltzoff, A. N. (1982) · Science · doi:10.1126/science.7146899 · Cited by 919 · see citing works

Perception and Production of Fluent Speech

Cole, R. A. (2016) · Routledge eBooks · doi:10.4324/9781315638935 · Cited by 779 · see citing works

NETtalk: a parallel network that learns to read aloud

Sejnowski, T. J., & Rosenberg, C. (1988) · Cited by 671 · see citing works

Review of Neural Networks for Speech Recognition

Lippmann, R. P. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.1 · Cited by 478 · see citing works

Visual influences on speech perception processes

Macdonald, J. B., & McGurk, H. (1978) · Perception & Psychophysics · doi:10.3758/bf03206096 · Cited by 458 · see citing works

Adaptive optical networks using photorefractive crystals

Psaltis, D., Brady, D. J., & Wagner, K. H. (1988) · Applied Optics · doi:10.1364/ao.27.001752 · Cited by 453 · see citing works

Some preliminaries to a comprehensive account of audio-visual speech perception.

Summerfield, Q. A. (1987) · Cited by 417 · see citing works

Confusions Among Visually Perceived Consonants

Fisher, C. G. (1968) · Journal of Speech and Hearing Research · doi:10.1044/jshr.1104.796 · Cited by 339 · see citing works

Automatic lipreading to enhance speech recognition (speech reading)

Petajan, E. D. (1984) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 328 · see citing works

Hearing by Eye: The Psychology of Lip-Reading

Repp, B. H., Dodd, B. J., & Campbell, R. (1988) · The American Journal of Psychology · doi:10.2307/1423237 · Cited by 320 · see citing works

Use of Visual Information for Phonetic Perception

Summerfield, Q. A. (1979) · Phonetica · doi:10.1159/000259969 · Cited by 320 · see citing works

Development of a Quantitative Description of Vowel Articulation

Stevens, K. N., & House, A. S. (1955) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907943 · Cited by 317 · see citing works

Large-vocabulary speaker-independent continuous speech recognition: the sphinx system

Reddy, R., & Lee, K.-F. (1988) · Cited by 310 · see citing works

Auditory-Visual Perception of Speech

Erber, N. P. (1975) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.4004.481 · Cited by 283 · see citing works

Optical Neural Computers

Abu‐Mostafa, Y. S., & Psaltis, D. (1987) · Scientific American · doi:10.1038/scientificamerican0387-88 · Cited by 256 · see citing works

Learned classification of sonar targets using a massively parallel network

Gorman, R. P., & B, T. J. S. A. (1988) · IEEE Transactions on Acoustics Speech and Signal Processing · doi:10.1109/29.1640 · Cited by 245 · see citing works

Effects of Training on the Visual Recognition of Consonants

Walden, B. E., Prosek, R. A., Montgomery, A. A., Scherr, C. K., & Jones, C. J. (1977) · Journal of Speech and Hearing Research · doi:10.1044/jshr.2001.130 · Cited by 218 · see citing works

The Development of an Experimental Discrete Dictation Recognizer

Jelinek, F. (1986) · Informatik-Fachberichte · doi:10.1007/978-3-642-71388-0_9 · Cited by 176 · see citing works

An improved automatic lipreading system to enhance speech recognition

Petajan, E. D., Bischoff, B., Bodoff, D., & Brooke, N. M. (1988) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/57167.57170 · Cited by 165 · see citing works

The development of an experimental discrete dictation recognizer

Jelinek, F. (1985) · Proceedings of the IEEE · doi:10.1109/proc.1985.13343 · Cited by 156 · see citing works

Introduction to Theoretical Linguistics

Potter, S., & Lyons, J. C. (1970) · The Modern Language Review · doi:10.2307/3723537 · Cited by 137 · see citing works

Physical characteristics of the lips underlying vowel lipreading performance

Montgomery, A. A., & Jackson, P. L. (1983) · The Journal of the Acoustical Society of America · doi:10.1121/1.389537 · Cited by 131 · see citing works

Volume holographic interconnections with maximal capacity and minimal cross talk

Lee, H., Gu, X.-G., & Psaltis, D. (1989) · Journal of Applied Physics · doi:10.1063/1.342828 · Cited by 127 · see citing works

The Calculation of Vowel Resonances, and an Electrical Vocal Tract

Dunn, H. K. (1950) · The Journal of the Acoustical Society of America · doi:10.1121/1.1906681 · Cited by 119 · see citing works

Auditory Tests with Synthetic Vowels

Miller, R. L. (1953) · The Journal of the Acoustical Society of America · doi:10.1121/1.1906983 · Cited by 93 · see citing works

Generating vocal tract shapes from formant frequencies

Ladefoged, P., Harshman, R. A., Goldstein, L. M., & Rice, L. (1978) · The Journal of the Acoustical Society of America · doi:10.1121/1.382086 · Cited by 89 · see citing works

A digital filter bank for spectral matching

Klatt, D. H. (2005) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1976.1170107 · Cited by 85 · see citing works

The sounds of speech communication : a primer of acoustic phonetics and speech perception

Pickett, J. M. (1980) · University Park Press eBooks · Cited by 78 · see citing works

Computer Recognition of Connected Speech

Reddy, D. R. (1967) · The Journal of the Acoustical Society of America · doi:10.1121/1.1910582 · Cited by 75 · see citing works

Single-channel vibrotactile supplements to visual perception of intonation and stress

Bernstein, L. E., Eberhardt, S. P., & Demorest, M. E. (1989) · The Journal of the Acoustical Society of America · doi:10.1121/1.397690 · Cited by 72 · see citing works

Lip Positions in American English Vowels

Fromkin, V. A. (1964) · Language and Speech · doi:10.1177/002383096400700402 · Cited by 62 · see citing works

Segmentation of Speech Sounds

Reddy, D. R. (1966) · The Journal of the Acoustical Society of America · doi:10.1121/1.1910071 · Cited by 42 · see citing works

A Comparative Analysis of the Audiovisual, Auditive and Visual Perception of Speech

Ewertsen, H. W., & Nielsen, H. B. (1971) · Acta Oto-Laryngologica · doi:10.3109/00016487109122473 · Cited by 30 · see citing works

Perceptual Dimensions Underlying Vowel Lipreading Performance

Jackson, P. L., Montgomery, A. A., & Binnie, C. A. (1976) · Journal of Speech and Hearing Research · doi:10.1044/jshr.1904.796 · Cited by 29 · see citing works

A perspective on man-machine communication by speech

Allen, J. (1985) · Proceedings of the IEEE · doi:10.1109/proc.1985.13339 · Cited by 13 · see citing works

Optical Implementation of Neural Computers

Psaltis, D., Brady, D. J., Qu, X.-G., & Hsu, K. Y. (1989) · Elsevier eBooks · doi:10.1016/b978-0-12-064470-4.50012-0 · Cited by 11 · see citing works

Structural analysis of lip-contours for isolated spoken vowels using fourier descriptors

Hesselmann, N. L. (1983) · Speech Communication · doi:10.1016/0167-6393(83)90049-3 · Cited by 6 · see citing works

Multimodal Machine Learning: A Survey and Taxonomy

Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works

Efficient Low-rank Multimodal Fusion With Modality-Specific Factors

Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · see citing works

Multimodal learning with transformers: A survey

Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works

Extraction of visual features for lipreading

Matthews, I. A., Cootes, T. F., Bangham, J., Cox, S. J., & Harvey, R. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982900 · Cited by 538 · see citing works

Review of Neural Networks for Speech Recognition

Lippmann, R. P. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.1 · Cited by 478 · see citing works

"Eigenlips" for robust speech recognition

Bregler, C., & Konig, Y. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1994.389567 · Cited by 312 · see citing works

Audio-visual integration in multimodal communication

Chen, T., & Rao, R. R. (1998) · Proceedings of the IEEE · doi:10.1109/5.664274 · Cited by 252 · see citing works

Audiovisual speech processing

Chen, T. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911195 · Cited by 231 · see citing works

Deep multimodal learning for Audio-Visual Speech Recognition

Mroueh, Y., Marcheret, E., & Goel, V. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178347 · Cited by 226 · see citing works

A Review on Methods and Applications in Multimodal Deep Learning

Jabeen, S., Li, X., Amin, M. S., Bourahla, O. E. F., Li, S., & Jabbar, A. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3545572 · Cited by 211 · see citing works

Effective Techniques for Multimodal Data Fusion: A Comparative Analysis

Pawłowski, M., Wróblewska, A., & Sysko-Romańczuk, S. (2023) · Sensors · doi:10.3390/s23052381 · Cited by 180 · see citing works

Nonlinear manifold learning for visual speech recognition

Bregler, C., & Omohundro, S. M. (2002) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1995.466899 · Cited by 150 · see citing works

Image sequence coding at very low bit rates: a review

Li, H., Lundmark, A., & Forchheimer, R. (1994) · IEEE Transactions on Image Processing · doi:10.1109/83.334983 · Cited by 143 · see citing works

Lipreading from color video

Chiou, G. I., & Hwang, J. (1997) · IEEE Transactions on Image Processing · doi:10.1109/83.605417 · Cited by 139 · see citing works

Audiovisual Information Fusion in Human–Computer Interfaces and Intelligent Environments: A Survey

Shivappa, S. T., Trivedi, M. M., & Rao, B. D. (2010) · Proceedings of the IEEE · doi:10.1109/jproc.2010.2057231 · Cited by 127 · see citing works

A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications

Jiao, T., Guo, C., Feng, X., Chen, Y., & Song, J. (2024) · Computers, Materials & Continua · doi:10.32604/cmc.2024.053204 · Cited by 158 · see citing works

The challenge of spoken language systems: Research directions for the nineties

Cole, R. A., Hirschman, L., Atlas, L., Beckman, M. E., Biermann, A. W., Bush, M. A., Clements, M. A., Cohen, L., Garcia, O., Hanson, B. A., et al. (1995) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.365385 · Cited by 125 · see citing works

Speechreading using Probabilistic Models

Luettin, J., & Thacker, N. A. (1997) · Computer Vision and Image Understanding · doi:10.1006/cviu.1996.0570 · Cited by 123 · see citing works

Improving connected letter recognition by lipreading

Bregler, C., Hild, H., Manke, S., & Waibel, A. H. (1993) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1993.319179 · Cited by 123 · see citing works

Design of a hybrid NAR-RBFs neural network for nonlinear dusty plasma system

Bukhari, A. H., Sulaiman, M., Raja, M. A. Z., Islam, S., Shoaib, M., & Kumam, P. (2020) · Alexandria Engineering Journal · doi:10.1016/j.aej.2020.04.051 · Cited by 113 · see citing works

Neural network lipreading system for improved speech recognition

Stork, D. G., Wolff, G. J., & Levine, E. (2003) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1992.226994 · Cited by 108 · see citing works

Continuous Automatic Speech Recognition by Lipreading

Goldschen, A. J., Garcia, O. N., & Petajan, E. D. (1997) · Computational imaging and vision · doi:10.1007/978-94-015-8935-2_14 · Cited by 107 · see citing works

Converting speech into lip movements: a multimedia telephone for hard of hearing people

Lavagetto, F. (1995) · IEEE Transactions on Rehabilitation Engineering · doi:10.1109/86.372898 · Cited by 107 · see citing works

Machine Learning as a Tool for Hypothesis Generation

Ludwig, J., & Mullainathan, S. (2024) · The Quarterly Journal of Economics · doi:10.1093/qje/qjad055 · Cited by 104 · see citing works

Visual speech recognition using active shape models and hidden Markov models

Luettin, J., Thacker, N. A., & Beet, S. W. (1996) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543246 · Cited by 104 · see citing works

Deep Multimodal Neural Architecture Search

Zhou, Y., Cui, Y., Yu, J., Wang, M., Tao, D., & Tian, Q. C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413977 · Cited by 103 · see citing works

Comparative analysis on cross-modal information retrieval: A review

Kaur, P., Pannu, H. S., & Malhi, A. K. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100336 · Cited by 97 · see citing works

Comparing models for audiovisual fusion in a noisy-vowel recognition task

Teissier, P., Robert-Ribes, J., Schwartz, J., & Guérin-Dugué, A. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.799688 · Cited by 94 · see citing works

Locally Confined Modality Fusion Network With a Global Perspective for Multimodal Human Affective Computing

Mai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89 · see citing works

Multimodality in meta-learning: A comprehensive survey

Ma, Y., Zhao, S., Wang, W., Li, Y., & King, I. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108976 · Cited by 89 · see citing works

Artificial Neural Networks and Genetic Algorithms

Cellier, F. E. (1991) · Springer eBooks · doi:10.1007/978-1-4757-3922-0_14 · Cited by 81 · see citing works

Multimedia Intelligence: When Multimedia Meets Artificial Intelligence

Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 80 · see citing works

Efficient Low-rank Multimodal Fusion with Modality-Specific FactorsPreprint

Liu, Z., Shen, Y., Lakshminarasimhan, V., Liang, P. P., Zadeh, A., & Morency, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00064 · Cited by 80 · see citing works

Automatic Speechreading with Applications to Human-Computer Interfaces

Zhang, X., Broun, C. C., Mersereau, R. M., & Clements, M. A. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206137 · Cited by 79 · see citing works

A taxonomy of Multimodal Interaction in the Human Information Processing System

Schomaker, L. (1995) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 72 · see citing works

Internet of medical things and trending converged technologies: A comprehensive review on real-time applications

Wagan, S. A., Koo, J., Siddiqui, I. F., Attique, M., Shin, D. R., & Qureshi, N. M. F. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.09.005 · Cited by 76 · see citing works

Speechreading using shape and intensity information

Luettin, J., Thacker, N. A., & Beet, S. W. (1996) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607024 · Cited by 70 · see citing works

A survey of identity recognition via data fusion and feature learning

Qin, Z., Zhao, P., Zhuang, T., Deng, F., Ding, Y., & Chen, D.-J. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.10.032 · Cited by 66 · see citing works

Multimodal interfaces

Waibel, A., Vo, M. T., Duchnowski, P., & Manke, S. (1996) · Artificial Intelligence Review · doi:10.1007/bf00127684 · Cited by 69 · see citing works

A Multimodal Data Processing System for LiDAR-Based Human Activity Recognition

Roche, J., Silva, V. D., Hook, J., Moencks, M., & KONDOZ, A. M. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3085489 · Cited by 68 · see citing works

Disaster Image Classification by Fusing Multimodal Social Media Data

Zou, Z., Gan, H., Huang, Q., Cai, T., & Cao, K. (2021) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi10100636 · Cited by 69 · see citing works

Analysis of multimodal data fusion from an information theory perspective

Dai, Y., Yan, Z., Cheng, J., Duan, X., & Wang, G. (2022) · Information Sciences · doi:10.1016/j.ins.2022.12.014 · Cited by 69 · see citing works

Recent Advances and Trends in Multimodal Deep Learning: A ReviewPreprint

Jabeen, S., Li, X., Amin, M. S., Li, S., & Abdul, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11087 · Cited by 88 · see citing works

A review of aquaculture: From single modality analysis to multimodality fusion

Li, W., Xu, X., Xu, X., Bai, Z., Han, J., Cui, M., & Li, D. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109367 · Cited by 60 · see citing works

Recognition of isolated words using Zernike and MFCC features for audio visual speech recognition

Borde, P. L., Varpe, A. B., Manza, R. R., & Yannawar, P. L. (2014) · International Journal of Speech Technology · doi:10.1007/s10772-014-9257-1 · Cited by 54 · see citing works

Adaptive Fusion of Speech and Lip Information for Robust Speaker Identification

Wark, T., & Sridharan, S. (2001) · Digital Signal Processing · doi:10.1006/dspr.2001.0397 · Cited by 53 · see citing works

Lip contour extraction from color images using a deformable model

Liew, A. W., Leung, S. H., & Lau, W. H. (2002) · Pattern Recognition · doi:10.1016/s0031-3203(01)00231-x · Cited by 52 · see citing works

Multimodal Fusion Method Based on Self-Attention Mechanism

Zhu, H., Wang, Z., Shi, Y., Hua, Y., Xu, G., & Deng, L. (2020) · Wireless Communications and Mobile Computing · doi:10.1155/2020/8843186 · Cited by 50 · see citing works

MPEG-4: Audio/video and synthetic graphics/audio for mixed media

Doenges, P. K., Çapın, T., Lavagetto, F., Östermann, J., Pandzic, I. S., & Petajan, E. D. (1997) · Signal Processing Image Communication · doi:10.1016/s0923-5965(97)00007-6 · Cited by 49 · see citing works

Recognition of Visual Speech Elements Using Adaptively Boosted Hidden Markov Models

Foo, S. W., Lian, Y., & Dong, L. (2004) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2004.826773 · Cited by 44 · see citing works

Statistical multimodal integration for audio-visual speech processing

Nakamura, S. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1021886 · Cited by 42 · see citing works

Multimodal Deep Learning Crime Prediction Using Tweets

Sakirin, T., & Tanrıöver, Ö. Ö. (2023) · IEEE Access · doi:10.1109/access.2023.3308967 · Cited by 40 · see citing works

End-to-end Multimodal Speech Recognition

Palaskar, S., Sanabria, R., & Metze, F. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462439 · Cited by 39 · see citing works

Visual features for context-aware speech recognition

Gupta, A., Miao, Y., Neves, L., & Metze, F. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953112 · Cited by 37 · see citing works

Hadamard Product in Deep Learning: Introduction, Advances and Challenges

Chrysos, G. G., Wu, Y., Pascanu, R., Torr, P. H. S., & Cevher, V. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3560423 · Cited by 37 · see citing works

Locating and tracking facial speech features

Luettin, J., Thacker, N. A., & Beet, S. W. (1996) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1996.546105 · Cited by 35 · see citing works

Multi-Modal Graph Interaction for Multi-Graph Convolution Network in Urban Spatiotemporal Forecasting

Zhang, L., Xu, G., Qin, Z. T., Wang, H., Wang, X., Zhang, Y., Liang, J., Wu, G., Song, X., & Wang, Y. (2022) · Sustainability · doi:10.3390/su141912397 · Cited by 34 · see citing works

Multimodal Conversational AI: A Survey of Datasets and Approaches

Sundar, A., & Heck, L. P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.12 · Cited by 37 · see citing works

Audio-Visual Transformer Based Crowd Counting

Sajid, U., Chen, X., Sajid, H., Kim, T., & Wang, G. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00254 · Cited by 32 · see citing works

Intra- and Inter-Modal Curriculum for Multimodal Learning

Zhou, Y., Wang, X. Y., Chen, H., Duan, X., & Zhu, W. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612468 · Cited by 28 · see citing works

Audio-Visual Model Distillation Using Acoustic Images

Pérez, A. F., Sanguineti, V., Morerio, P., & Murino, V. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093307 · Cited by 27 · see citing works

End-to-end Neuromorphic Lip Reading

Bulzomi, H., Schweiker, M., Gruel, A., & Martinet, J. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00431 · Cited by 27 · see citing works

Detecting audio-visual synchrony using deep neural networks

Marcheret, E., Potamianos, G., Vopicka, J., & Goel, V. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-201 · Cited by 25 · see citing works

Open-Domain Audio-Visual Speech Recognition: A Deep Learning Approach

Miao, Y., & Metze, F. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-412 · Cited by 25 · see citing works

A hybrid approach to bimodal speech recognition

Bregler, C., Omohundro, S. M., & Konig, Y. (2002) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/acssc.1994.471514 · Cited by 25 · see citing works

Time-delay neural networks for estimating lip movements from speech analysis: a useful tool in audio-video synchronization

Lavagetto, F. (1997) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/76.633499 · Cited by 25 · see citing works

Image Sequence Coding at Verv Low Bitrates: A Review

Li, H., Lundmark, A., & Forchheimer, R. (1994) · Cited by 25 · see citing works

Deep Multimodal Learning for Audio-Visual Speech RecognitionPreprint

Mroueh, Y., Marcheret, E., & Goel, V. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.05396 · Cited by 25 · see citing works

Dual Low-Rank Multimodal Fusion

Jin, T., Huang, S., Li, Y., & Zhang, Z. M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.35 · Cited by 24 · see citing works

A comparison of active shape model and scale decomposition based features for visual speech recognition

Matthews, I. A., Bangham, J., Harvey, R., & Cox, S. J. (1998) · Lecture notes in computer science · doi:10.1007/bfb0054762 · Cited by 24 · see citing works

Multimodal Relation Extraction via a Mixture of Hierarchical Visual Context Learners

Liu, X., Hu, C., Zhang, R., Sun, K., Mensah, S., & Mao, Y. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645603 · Cited by 23 · see citing works

A comparison of models for fusion of the auditory and visual sensors in speech perception

Robert-Ribes, J., Schwartz, J., & Escudier, P. (1995) · Artificial Intelligence Review · doi:10.1007/bf00849043 · Cited by 23 · see citing works

Fusion of Audio-Visual Information for Integrated Speech Processing

Nakamura, S. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45344-x_20 · Cited by 22 · see citing works

Deep Tensor Evidence Fusion Network for Sentiment Classification

Wang, Z., Xu, G., Zhou, X., Kim, J., Zhu, H., & Deng, L. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3197994 · Cited by 21 · see citing works

Multi-task Hierarchical Heterogeneous Fusion Framework for multimodal summarization

Zhang, L., Zhang, X., Han, L., Yu, Z., Liu, Y., & Li, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103693 · Cited by 21 · see citing works

Label distribution for multimodal machine learning

Ren, Y. C., Xu, N., Ling, M., & Geng, X. (2021) · Frontiers of Computer Science · doi:10.1007/s11704-021-0611-6 · Cited by 21 · see citing works

Fuzzy measures in inductive reasoning

Li, D., & Cellier, F. E. (1990) · NCSU Libraries Repository (North Carolina State University Libraries) · doi:10.5555/328885.329028 · Cited by 21 · see citing works

A Self-Organized Artificial Neural Network Architecture for Sensory Integration with Applications to Letter-Phoneme Integration

Jantvik, T., Gustafsson, L., & Papliński, A. P. (2011) · Neural Computation · doi:10.1162/neco_a_00149 · Cited by 20 · see citing works

A multimodal deep architecture for traditional Chinese medicine diagnosis

Dai, Y., Wang, G., Dai, J., & Geman, O. (2020) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.5781 · Cited by 19 · see citing works

An overview of protected satellite communications in intelligent age

Wang, C., Zhang, Z., Wu, J., Chen, C., & Gao, F. (2021) · Science China Information Sciences · doi:10.1007/s11432-019-2928-9 · Cited by 19 · see citing works

Multimodal architecture for emotion in robots using deep learning

Ghayoumi, M., & Bansal, A. K. (2016) · Future Technologies Conference · doi:10.1109/ftc.2016.7821710 · Cited by 19 · see citing works

Bimodal sensor integration on the example of 'speechreading'

Bregler, C., Manke, S., Hild, H., & Waibel, A. H. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1993.298634 · Cited by 19 · see citing works

Information fusion in sea ice remote sensing

Collins, M. J. (1992) · Geophysical monograph · doi:10.1029/gm068p0431 · Cited by 17 · see citing works

Combining Visual and Acoustic Speech Signals with a Neural Network Improves Intelligibility

B, T. J. S. A., Yuhas, B. P., Goldstein, M. H., & Jenkins, R. E. (1989) · neural information processing systems · Cited by 17 · see citing works

A defensive attention mechanism to detect deepfake content across multiple modalities

Asha, S., Vinod, P., & Menon, V. G. (2024) · Multimedia Systems · doi:10.1007/s00530-023-01248-x · Cited by 16 · see citing works

Audio and visual modality combination in speech processing applications

Potamianos, G., Marcheret, E., Mroueh, Y., Goel, V., Koumbaroulis, A., Vartholomaios, A., & Thermos, S. (2017) · ACM eBooks · doi:10.1145/3015783.3015797 · Cited by 16 · see citing works

Vowel Recognition using Neural Networks

Sadeghi, V. S., & Yaghmaie, K. (2006) · Cited by 16 · see citing works

A Two-Channel Training Algorithm for Hidden Markov Model and Its Application to Lip Reading

Dong, L., Foo, S. W., & Lian, Y. (2005) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/asp.2005.1382 · Cited by 16 · see citing works

Fuzzy measures in inductive reasoning

Li, D., & Cellier, F. E. (2002) · Winter Simulation Conference · doi:10.1109/wsc.1990.129571 · Cited by 16 · see citing works

Object-oriented scene modeling for interpersonal video communication at very low bit-rate

Lavagetto, F., & Curinga, S. (1994) · Signal Processing Image Communication · doi:10.1016/0923-5965(94)90001-9 · Cited by 16 · see citing works

Multimodal Interfaces

Waibel, A., Vo, M. T., Duchnowski, P., & Manke, S. (1996) · Springer eBooks · doi:10.1007/978-94-009-1716-3_9 · Cited by 16 · see citing works

Advancing classroom fatigue recognition: A multimodal fusion approach using self-attention mechanism

Cao, L., Wang, W., Dong, Y., & Fan, C. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105756 · Cited by 15 · see citing works

Enhancing 3D Reconstruction Accuracy of FIB Tomography Data Using Multi-voltage Images and Multimodal Machine Learning

Sardhara, T., Shkurmanov, A., Li, Y., Riedel, L., Shi, S., Cyron, C. J., Aydin, R. C., & Ritter, M. (2024) · Nanomanufacturing and Metrology · doi:10.1007/s41871-024-00223-y · Cited by 15 · see citing works

Hyper column model vs. fast DCT for feature extraction in visual arabic speech recognition

Sagheer, A., Tsuruta, N., Taniguchi, R., & Maeda, S. (2006) · IEEE International Symposium on Signal Processing and Information Technology (ISSPIT) · doi:10.1109/isspit.2005.1577194 · Cited by 15 · see citing works

Weakly Supervised Representation Learning for Unsynchronized Audio-Visual EventsPreprint

Parekh, S., Essid, S., Ozerov, A., Duong, N. Q. K., Pérez, P., & Gaël Richard, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.07345 · Cited by 15 · see citing works

A survey of automatic lip reading approaches

Butt, W. U. R., & Lombardi, L. (2013) · International Conference on Digital Information Management · doi:10.1109/icdim.2013.6694023 · Cited by 14 · see citing works

A review on large language models for industrial embodied intelligence

Zhu, J., Huang, S., Wang, P., Xu, Z., Liu, J., Wang, B., Zhao, Z., Zheng, S., Tao, Y., Wang, G., et al. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.104602 · Cited by 13 · see citing works

Understanding Political Polarization via Jointly Modeling Users, Connections and Multimodal Contents on Heterogeneous Graphs

Lyu, H., & Luo, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547898 · Cited by 13 · see citing works

Multi-modal cognitive computing

李, 学. (2022) · Scientia Sinica Informationis · doi:10.1360/ssi-2022-0226 · Cited by 13 · see citing works

Enhancing automatic speech recognition with an ultrasonic lip motion detector

Jennings, D. L., & Ruck, D. W. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1995.479832 · Cited by 13 · see citing works

Combining Visual and Acoustic Speech Signals with a Neural Network Improves Intelligibility

B, T. J. S. A., Yuhas, B. P., Goldstein, M. H., & Jenkins, R. E. (1989) · neural information processing systems · Cited by 17 · see citing works

Interpreting visual speech signals using neural networks

Yuhas, B. P. (1989) · The Journal of the Acoustical Society of America · doi:10.1121/1.2026759

The processing of visual speech signals using parallel distributed computation

Yuhas, B. P. (1990)

Neural network models of sensory integration for improved vowel recognition

Yuhas, B. P., Goldstein, M. H., B, T. J. S. A., & Jenkins, R. E. (1990) · Proceedings of the IEEE · doi:10.1109/5.58349 · Cited by 80 · see citing works

Massively parallel network architectures for automatic recognition of visual speech signals. Final technical report

B, T. J. S. A., & Goldstein, M. M. (1990) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information)