Integration of acoustic and visual speech signals using neural networks

Yuhas, B.P., Goldstein, M.H., Sejnowski, T.J.·1989·IEEE Communications Magazine
DOI 10.1109/35.41402

Cited by (209)

Items 101–200 of 209

  1. Models for Audiovisual Fusion in a Noisy-Vowel Recognition Task
    Pascal Teissier, Anne Guérin-Dugué, Jean‐Luc Schwartz · 1998 · The Journal of VLSI Signal Processing Systems for Signal Image and Video Technology
  2. Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?
    Jielin Qiu, William Han, Jiacheng Zhu, Mengdi Xu, Michael Rosenberg, Emerson Liu, Douglas J. Weber, Ding Xuan Zhao · 2023 · Findings of the Association for Computational Linguistics: EACL
  3. Two-Dimensional Attentive Fusion for Multi-Modal Learning of Neuroimaging and Genomics Data
    Md Abdur Rahaman, Yash Garg, Armin Iraji, Zening Fu, Jiayu Chen, Vince D. Calhoun · 2022 · Machine learning for signal processing ...
  4. A Novel Multi-Modal One-Shot Learning Method for Texture Recognition
    Pengwen Xiong, Kongfei He, Aiguo Song, Peter Xiaoping Liu · 2019 · IEEE Access
  5. Complementary information mutual learning for multimodality medical image segmentation
    Chuyun Shen, Wenhao Li, Haoqing Chen, Xiaoling Wang, Fengping Zhu, Yuxin Li, Xiangfeng Wang, Bo Jin · 2024 · Neural Networks
  6. Depression detection in read and spontaneous speech: A Multimodal approach for lesser-resourced languages
    Klara Daly, Oluwafemi Olukoya · 2025 · Biomedical Signal Processing and Control
  7. Audio-Visual Speech Processing for Human Computer Interaction
    Siew Wen Chin, Kah Phooi Seng, Li-Minn Ang · 2011 · Intelligent systems reference library
  8. Audio-visual interaction in multimodal communication
    Rama Chellappa, Tsuhan Chen, Aggelos K. Katsaggelos · 1997 · IEEE Signal Processing Magazine
  9. What can visual speech synthesis tell visual speech recognition?
    M.M. Cohen, Dominic W. Massaro · 2002 · Asilomar Conference on Signals, Systems and Computers (ACSSC)
  10. Speechreading using shape and intensity information
    Juergen Luettin, N. A. Thacker, Steve W. Beet · 1996 · International Conference on Spoken Language Processing (ICSLP)
  11. Multimodal Machine Learning for Real Estate Appraisal: A Comprehensive Survey
    Chenya Huang, Bin Liang, Zhidong Li, Fang Chen · 2025 · Lecture notes in computer science
  12. AviPer: assisting visually impaired people to perceive the world with visual-tactile multimodal attention network
    Xinrong Li, Meiyu Huang, Yao Xu, Yingze Cao, Yamei Lu, Pengfei Wang, Xueshuang Xiang · 2022 · CCF Transactions on Pervasive Computing and Interaction
  13. Arabic Lip-reading System: A Combination of Hypercolumn Neural Network Model with Hidden Markov Model
    Alaa Sagheer, Naoyuki Tsuruta, 直之 鶴田, Rin-ichiro Taniguchi, 倫一郎 谷口, サギール アラー, ナオユキ ツルタ, リンイチロウ タニグチ · 2004 · Kyushu University Institutional Repository (QIR) (Kyushu University)
  14. On the active perception of speech by robots
    Harouna Kabré · 2002 · IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)
  15. Sparse Fusion for Multimodal Transformers
    Yi Ding, Alex Rich, Mason Long Wang, Noah Stier, Matthew Turk, Pradeep Sen, Tobias Höllerer · 2022
  16. Algorithmic Behavioral Science: Machine Learning as a Tool for Scientific Discovery
    Jens Ludwig, Sendhil Mullainathan · 2022 · SSRN Electronic Journal
  17. Semantic fusion of facial expressions and textual opinions from different datasets for learning-centered emotion recognition
    Héctor Manuel Cárdenas López, Ramón Zataraín Cabada, María Lucía Barrón Estrada, Hugo A. Mitre-Hernández · 2023 · Soft Computing
  18. A Simple Analysis of Multimodal Data Fusion
    Jiangchang Cheng, Yinglong Dai, Yuan Yao, Hongli Zhu · 2020 · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom)
  19. Sensor Fusion and Multimodal Learning for Robotic Grasp Verification Using Neural Networks
    Priteshkumar Gohil, Santosh Thoduka, Paul G. Plöger · 2022 · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition
  20. Multimodal Fast–Slow Neural Network for learning engagement evaluation
    Lizhao Zhang, Jui-Long Hung, Xu Du, Hao Li, Zhuang Hu · 2023 · Data Technologies and Applications
  21. Automatic visual speech segmentation and recognition using directional motion history images and Zernike moments
    Ayaz A. Shaikh, Dinesh Kumar, Jayavardhana Gubbi · 2012 · The Visual Computer
  22. Classification of visemes using visual cues
    Nazeeh Alothmany, Robert Boston, Ching Lin Li, Susan Shaiman, John D. Durrant · 2009 · D-Scholarship@Pitt (University of Pittsburgh)
  23. Automatic combined lip segmentation in color images
    Hamed Talea, Khashayar Yaghmaie · 2011 · IEEE International Conference on Communication Software and Networks
  24. Fusion of visual and acoustic signals for command-word recognition
    R. Kober, Ulrich Harz, Jutta Schiffers · 2002 · IEEE International Conference on Acoustics Speech and Signal Processing
  25. ViST: A Ubiquitous Model with Multimodal Fusion for Crop Growth Prediction
    Junsheng Li, Ling Wang, Jie Liu, Jinshan Tang · 2023 · ACM Transactions on Sensor Networks
  26. Audio–visual self-supervised representation learning: A survey
    Manal S. Alsuwat, Sarah Al-Shareef, Manal Alghamdi · 2025 · Neurocomputing
  27. From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation
    Dhruv Agarwal, Tanay Agrawal, Laura M. Ferrari, François Brémond · 2021 · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)
  28. Audio-visual speech recognition by speechreading
    Xiaozheng Zhang, Russell M. Mersereau, Mark A. Clements · 2003 · IEEE International Conference on Digital Signal Processing (DSP)
  29. Synthesis and animation of human faces: artificial reality in interpersonal video communication
    Sergio Curinga, Aldo Grattarola, Fabio Lavagetto · 1993 · Springer eBooks
  30. Evolution, learning and speech recognition in changing acoustic environments
    Anne Spalanzani, Harouna Kabré · 1998 · Lecture notes in computer science
  31. Neural processing-type displacement sensor employing multimode waveguide
    Shigeki Aisawa, Kazuhiro Noguchi, T. Matsumoto · 1991 · IEEE Photonics Technology Letters
  32. Direct identification vs. correlated models to process acoustic and articulatory informations in automatic speech recognition
    Régine Andre-Obrecht, Bruno Jacob · 2002 · IEEE International Conference on Acoustics Speech and Signal Processing
  33. Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering
    Qiwen Lu, Shengbo Chen, Xiaoke Zhu · 2024 · Journal of Imaging
  34. Vowel recognition of patients after total laryngectomy using Mel Frequency Cepstral Coefficients and mouth contour
    Rafał Pietruch, Antoni D. Grzanka · 2010 · Journal of Automatic Control
  35. Lip Tracking Towards an Automatic Lip Reading Approach.
    Luca Lombardi, Waqqas ur Rehman Butt, Marco Grecuccio · 2014 · J. Multim. Process. Technol.
  36. Reconocimiento multimodal de emociones orientadas al aprendizaje
    Ramón Zataraín Cabada, María Lucía Barrón Estrada, Héctor Manuel Cárdenas López · 2019 · Research in Computing Science
  37. CNN: A speaker recognition system using a cascaded neural network
    Mervat S. Zaki, Asem Ahmed Ghalwash, A. A. Elkouny · 1996 · Multidimensional Systems and Signal Processing
  38. Detection of redundant frame in audio visual speech recognition using low level analysis
    Pravin Laxmikant Yannawar, Ganesh Manza, Bharti W. Gawali, S. C. Mehrotra · 2010 · IEEE International Conference on Computational Intelligence and Computing Research (ICCIC)
  39. Models for audiovisual fusion in a noisy-vowel recognition task
    Pascal Teissier, Jean‐Luc Schwartz, Anne Guérin-Dugué · 2002 · IEEE International Workshop on Multimedia Signal Processing (MMSP)
  40. A neural clustering algorithm for estimating visible articulatory trajectory
    Fabio Vignoli, Sergio Curinga, Fabio Lavagetto · 1996 · Lecture notes in computer science
  41. A Multimodal Sensor Fusion Architecture for Audio-Visual Speech Recognition
    Mustapha Makkook · 2007 · UWSpace (University of Waterloo)
  42. Imaging‐genomic spatial‐modality attentive fusion for studying neuropsychiatric disorders
    Md Abdur Rahaman, Yash Garg, Armin Iraji, Zening Fu, Peter V Kochunov, Liyi Elliot Hong, Theo G.M. van Erp, Adrian Preda, Jiayu Chen, Vince D. Calhoun · 2024 · Human Brain Mapping
  43. Analysis of Induced Color for Automatic Detection of ROI in Multipose AVSR System
    Amarsinh Bhimrao Varpe, Prashant L. Borde, Sadhana Sukale, Pallavi Perdeshi, Pravin Laxmikant Yannawar · 2015 · Advances in intelligent systems and computing
  44. Review of Lip-Reading Recognition
    Guoli Zheng, Meili Zhu, Lili Feng · 2014 · International Symposium on Computational Intelligence and Design
  45. Fusion Architectures for Word-Based Audiovisual Speech Recognition
    Michael Wand, Jürgen Schmidhuber · 2020 · Interspeech (USB)
  46. Multiview Approaches to Event Detection and Scene Analysis
    Slim Essid, Sanjeel Parekh, Ngoc Q. K. Duong, Romain Serizel, Alexey Ozerov, Fabio Antonacci, Augusto Sarti · 2017 · Springer eBooks
  47. Muscle modeling for facial animation in videophone coding
    Carlo Braccini, Sergio Curinga, Aldo Grattarola, Fabio Lavagetto · 2003 · IEEE RO-MAN
  48. A Comprehensive Review of Multimodal Visual Representation Learning: Tracing the Evolution from CNNs to Transformers and Beyond
    Dong Zhang, Wei Kitt Wong, Ing Ming Chew · 2025 · International Journal of Multimedia Information Retrieval
  49. Large language model-enhanced graph neural network for quantile prediction of railway track settlement near deep excavations
    Xiaojing Zhou, Yue Pan, Jianjun Qin, Jin-Jian Chen · 2025 · Advanced Engineering Informatics
  50. CMAViT: Integrating Climate, Management, and Remote Sensing Data for Crop Yield Prediction With Multimodel Vision Transformers
    Hamid Kamangir, B. Sams, Nick Dokoozlian, Luis Sanchez, J. Mason Earles · 2026 · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
  51. Scattering vs. discrete cosine transform features in visual speech processing.
    Etienne Marcheret, Gerasimos Potamianos, Josef Vopicka, Vaibhava Goel · 2015 · AVSP
  52. Combining acoustic and visual modalities in vowel recognition system for laryngectomees
    Rafał Pietruch, Antoni D. Grzanka · 2010 · Symposium on Neural Network Applications in Electrical Engineering
  53. Fusing Visual and Textual Information to Determine Content Safety
    Rodrigo Leonardo, Amber Hu, Mohammad Uzair, Qiujing Lu, Iris Fu, Keishin Nishiyama, Sooraj Mangalath Subrahmannian, Divyaa Ravichandran · 2019 · International Conference on Machine Learning and Applications
  54. Learning Fuzzy Rules for Visual Speech Recognition
    Mohd Ayaan Anwar, Jim F. Baldwin, Trevor Martin · 2004 · Lecture notes in computer science
  55. A survey on multimodal explainable Artificial Intelligence
    Claudio Giovannoni, Carlo Metta, Anna Monreale, Salvatore Rinzivillo · 2026 · Intelligent Systems with Applications
  56. Multimodal machine learning in real estate appraisal: models, data, and trends
    Chenya Huang, Bin Liang, Zhidong Li, Justin Wang, Fang Chen · 2026 · International Journal of Data Science and Analytics
  57. Augmented classification of Japanese visemes and hierarchical weighted discrimination for visual speech recognition
    Shinsuke Okita, Yasue Mitsukura, Nozomu Hamada · 2013 · IEEE Conference on Systems, Process and Control
  58. A Fusion Approach for Kannada Speech Recognition Using Audio and Visual Cue
    R Shashidhar, M. P. Shashank, G. Jagadamba, Vinayakumar Ravi · 2024 · Smart sensors, measurement and instrumentation
  59. Speaker Recognition Based on Lip-reading
    Dongpeng Shang, Xinman Zhang, Xuebin Xu, Xiangyang Peng · 2018 · International Conference Network, Communication and Computing
  60. Detecting Multi Thoracic Diseases in Chest X-Ray Images Using Deep Learning Techniques
    Sebastián Quevedo, Federico Domínguez, Enrique Peláez · 2023 · International Conference on Pattern Recognition Systems (ICPRS)
  61. Audio-visual speech recognition for a vowel discrimination task
    Peter L. Silsbee, Alan Conrad Bovik · 1993 · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE
  62. Robustness of a chaotic modal neural network applied to audio-visual speech recognition
    Harouna Kabré · 2002 · Machine learning for signal processing ...
  63. Audio-Visual Transformer Based Crowd Counting
    Usman Sajid, Xiangyu Chen, Hasan Sajid, Taejoon Kim, Guanghui Wang · 2021 · arXiv (Cornell University)
  64. A Comparison of Models for Fusion of the Auditory and Visual Sensors in Speech Perception
    Jordi Robert-Ribes, Jean‐Luc Schwartz, Pierre Escudier · 1995 · Springer eBooks
  65. Lip reading system using novel Japanese visemes classification and hierarchical weighted discrimination
    Shinsuke Okita, Yasue Mitsukura, Nozomu Hamada · 2013 · International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS)
  66. The aesthetic nature of Chinese-inscribed poetry and painting texts—evidence from eye movements
    Huang Yi, Weiran Rong, Xiang Robert Li, Wenyu Wu, Yi Liu, Shiyu Sun, Feng Gan · 2025 · Digital Scholarship in the Humanities
  67. Robust defense strategies for multimodal contrastive learning: efficient fine-tuning against backdoor attacks
    Md. Iqbal Hossain, Afia Sajeeda, Neeresh Kumar Perla, Ming Shao · 2026 · Multimedia Tools and Applications
  68. Integrating Multimodal Learning and Explainable AI for Enhanced and Interpretable Prostate Lesion Classification
    Claudio Giovannoni, Carlo Metta, Andrea Berti, Sara Colantonio, Anna Monreale, Francesca Pratesi, Salvatore Rinzivillo · 2026 · Machine Learning
  69. Enhancing Robustness in Speech Recognition using Visual Information
    Omar Farooq, Sekharjit Datta · 2012 · IGI Global eBooks
  70. Sentiment Analysis of Telephone Conversations Using Multimodal Data
    Alexander Gafuanovich Logumanov, Julius Klenin, Dmitry Botov · 2018 · Lecture notes in computer science
  71. Chinese Tone Recognition Based on 3D Dynamic Muscle Information
    Jianrong Wang, Li Wan, Ju Zhang, Qiang Fang, Fan Yang, Jing Hu · 2020 · Discrete Dynamics in Nature and Society
  72. Multimodal Learning for Automatic Summarization: A Survey
    Zhicheng Zhang, Yibo Sun, Shiyan Su · 2023 · Lecture notes in computer science
  73. FlyTransformer: A Cross-Modal Fusion Policy for UAV End-to-End Trajectory Planning
    Wenxiang Shi, Chen Zhao, Kailei Tang, Junru Sheng, Zhiyan Dong, Lihua Zhang, Xiaoyang Kang, Kai Cao · 2023 · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics
  74. A Transformer Substation Defect Detection Method Based on Multimodal Knowledge Graph
    Zhenyu Guo, Tao Zhu, Daojun Huang · 2024 · International Conference on HVDC (HVDC)
  75. Automatic isolation of region of interest for multi-pose Audio Visual Speech Recognition
    Amarsinh Bhimrao Varpe, Prashant L. Borde, Ramesh R. Manza, Pravin Laxmikant Yannawar · 2014 · IEEE India Conference
  76. Audio-visual automatic speech recognition using Dynamic Bayesian Networks
    Helge Reikeras · 2011 · SUNScholar (Stellenbosch University)
  77. Cross-speaker viseme mapping using hidden Markov models
    Liang Dong, Say Wei Foo, Yong Lian · 2004 · International Conference on Information, Communications & Signal Processing (ICICS)
  78. Improving spoken term detection using complementary information
    Shahram Kalantari · 2015
  79. Learning deep representation of multityped objects and tasks
    Truyen Tran, Dinh Phung, Svetha Venkatesh · 2016 · arXiv (Cornell University)
  80. Somatic features extraction for facial mimics synthesis
    W. Bellotto, Sergio Curinga, P. Ghelfi, Fabio Lavagetto · 1994 · Elsevier eBooks
  81. Deep Cross-Modal Alignment in Audio-Visual Speech Recognition
    George Sterpu · 2021 · Trinity's Access to Research Output (TARA) (Trinity College Dublin)
  82. Artificial Intelligence Techniques for Analysis: Expert Systems and Neural Networks
    James H. Griesmer, James A. Kierstead, Michael Rothman · 1993 · Springer eBooks
  83. ASR Based on the Analasys of the Short-MelFrequencyCepstra Time Transform
    Juan A. Nolazco‐Flores · 2005 · Lecture notes in computer science
  84. Acoustic Adaptation in Cross Database Audio Visual SHMM Training for Phonetic Spoken Term Detection
    Shahram Kalantari, David Dean, Sridha Sridharan, Houman Ghaemmaghami, Clinton Fookes · 2015
  85. Visual speech signal representations according to an articulatory model based on X-ray motion films
    Wen Chengyi, Nan Chen, Zhang Yufu · 2003 · IAPR International Conference on Pattern Recognition (ICPR)
  86. A Novel Visual Feature Extraction and Its Application in Vowel Recognition
    Vahideh Sadat Sadeghi, Khashayar Yaghmaie · 2007 · IEEE International Conference on Data Engineering Workshops (ICDEW)
  87. Information theory applied to neural networks
    Qi Yu · 1993 · ThinkTech (Texas Tech University)
  88. Non-linear adaptive equalization based on a multi-layer perceptron architecture.
    Sammy Siu · 1991 · ERA
  89. A Combination of Hyper Column Model with Hidden Markov Model for Japanese Lip-Reading System
    Alaa Sagheer, Naoyuki Tsuruta, 直之 鶴田, Rin-ichiro Taniguchi, 倫一郎 谷口, Sakashi Maeda, 佐嘉志 前田, Seiji Hashimoto, 誠司 橋本, サギール アラー et al. · 2004 · Kyushu University Institutional Repository (QIR) (Kyushu University)
  90. A study on robust audio-visual speech recognition
    哲嗣 田村 · 2005
  91. Hidden Markov Model Based Visemes Recognition, Part I
    Say Wei Foo, Liang Donga · 2009 · IGI Global eBooks
  92. 7. Conclusion and Future Work
    2012 · De Gruyter eBooks
  93. Cross database audio visual speech adaptation for phonetic spoken term detection
    Shahram Kalantari, David Dean, Sridha Sridharan · 2017 · Computer Speech & Language
  94. Multi-signal processing for voice recognition in noisy environments
    Taysir H. Nayfeh · 1991 · VTechWorks (Virginia Tech)
  95. Speechreading using Modified Visual Feature Vectors
    Preety Singh, Vijay Laxmi, Manoj Singh Gaur · 2012 · IGI Global eBooks
  96. TCT: A Cross-supervised Learning Method for Multimodal Sequence Representation
    Wubo Li, Zou, Wei, Xiangang Li · 2019 · arXiv (Cornell University)
  97. A framework for comprehensible multi-modal detection of cyber threats
    Jan Kohout, Cenek Skarda, Kyrylo Shcherbin, Martin Kopp, Jan Brabec · 2021 · arXiv (Cornell University)
  98. Online video visual relation detection with hierarchical multi-modal fusion
    Yuxuan He, Ming-Gang Gan, Qianzhao Ma · 2024 · Multimedia Tools and Applications
  99. Multi-modal Spatial-modality Attentive Fusion for Studying Neuropsychiatric Disorders
    Md Abdur Rahaman, Yash Garg, Armin Iraji, Zening Fu, Peter V Kochunov, Liyi Elliot Hong, Theo G.M. van Erp, Adrian Preda, Jiayu Chen, Vince D. Calhoun · 2024 · bioRxiv (Cold Spring Harbor Laboratory)
  100. Pseudo-Label Guided Partially View-Aligned Clustering
    Songtao Wu, Ruixin Ma, Qiongjie Xie, Liang Zhao · 2024 · IEEE International Conference on Big Data Analytics (ICBDA)