The DOI matches its Crossref record; 31 of the 34 references it cites were found in a registry or an index.
Multimodal integration-a statistical view
Abstract
We present a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We first identify the primary factors that influence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estimate approach and a learning approach, which are designed to optimize accurate recognition during the multimodal integration process. We evaluate these methods using Quickset, a speech/gesture multimodal system, and report evaluation results based on an empirical corpus collected with Quickset. From an architectural perspective, the integration technique presented offers enhanced robustness. It also is premised on more realistic assumptions than previous multimodal systems using semantic fusion. From a methodological standpoint, the evaluation techniques that we describe provide a valuable tool for evaluating multimodal systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For instance, Masked Language Modelling (MLM) in the text domain has been applied to audio and image, e.g., Masked Acoustic Modelling [180], [200], Masked Image Region Prediction [190]”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [200]
Citations
Cites
Pattern classification and scene analysis
Grenander, U., Duda, R., & Hart, P. (1974) · A Wiley-Interscience publication · doi:10.2307/1573081 · Cited by 17,399
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,394
Bolt, R. A. (1980) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/800250.807503 · Cited by 1,493
Bridle, J. S. (1990) · Springer eBooks · doi:10.1007/978-3-642-76153-9_28 · Cited by 1,278
Bolt, R. A. (1980) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965105.807503 · Cited by 921
The logic of typed feature structures
Carpenter, B. (1992) · Cited by 600
Cohen, P. R., Johnston, M. O., McGee, D. R., Oviatt, S. L., Pittman, J., Smith, I., Chen, L., & Clow, J. (1997) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/266180.266328 · Cited by 466
Self-citationData fusion in robotics and machine intelligence
Abidi, M. A., & Gonzalez, R. C. (1992) · Academic Press Professional, Inc. eBooks · Cited by 444
The Logic of Typed Feature Structures
Carpenter, R. L. (1992) · Cambridge University Press eBooks · doi:10.1017/cbo9780511530098 · Cited by 410
Flagged first, then by citations
3 of the 34 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- poddar, 1998, toward natural gesture/speech hci: a case study of weather narration, Proc 1998 Workshop on Perceptual User Interfaces (PUI 98), 1
- wu, 0, From members to teams to committee— A robust approach to gestural and multimodal recognition
- young, 1996, Large vocabulary continuous speech recognition A review
Cited by
Sorted by citations
Items 101–190 of 190
Implementation testing of a hybrid symbolic/statistical multimodal architecture
Kaiser, E., & Cohen, P. R. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-113 · Cited by 10
Self-citationAn integrative recognition method for speech and gestures
Miki, M., Miyajima, C., Nishino, T., Kitaoka, N., & Takeda, K. (2008) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1452392.1452411 · Cited by 9
Semi-synchronous speech and pen input for mobile user interfaces
Shinoda, K., Watanabe, Y., Iwata, K., Liang, Y., Nakagawa, R., & Furui, S. (2010) · Speech Communication · doi:10.1016/j.specom.2010.10.001 · Cited by 9
Invariant Object Recognition Robot Vision System for Assembly
Peña‐Cabrera, M., López-Juárez, I., & Osorio-Comparán, R. (2006) · Electronics, Robotics and Automotive Mechanics Conference · doi:10.1109/cerma.2006.53 · Cited by 9
Speech Communication and Multimodal Interfaces
Schuller, B. W., Ablaßmeier, M., Müller, R., Reifinger, S., Poitschke, T., & Rigoll, G. (2006) · Springer eBooks · doi:10.1007/3-540-30619-6_4 · Cited by 9
From Monomodal to Multimodal: Affect Recognition Using Visual Modalities
Monekosso, D., Remagnino, P., & Kuno, Y. (2008) · Intelligent Environments · doi:10.1007/978-1-84800-346-0_10 · Cited by 9
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality
Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P., & Feiner, S. (2003) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/958436.958438 · Cited by 9
Self-citationInteractive Exploration in Virtual Environments
Belleman, R. G. (2003) · Data Archiving and Networked Services (DANS) · Cited by 9
Optimality and Limitations of Audio-Visual Integration for Cognitive Systems
Boyce, W. P., Lindsay, A., Zgonnikov, A., Rañó, I., & Wong‐Lin, K. (2020) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.00094 · Cited by 11
Survey of Data Mining Methods in Emergency Evacuation Planning
Miah, M. (2011) · Cited by 8
Multimodal speech and pen interfaces
Cohen, P. R., & Oviatt, S. L. (2017) · ACM eBooks · doi:10.1145/3015783.3015795 · Cited by 8
Self-citationMultimodal gesture recognition
Katsamanis, A., Pitsikalis, V., Theodorakis, S., & Maragos, P. A. (2017) · ACM eBooks · doi:10.1145/3015783.3015796 · Cited by 8
Statistical multimodal integration for intelligent HCI
Wu, L., Oviatt, S. L., & Cohen, P. R. (2003) · Machine learning for signal processing ... · doi:10.1109/nnsp.1999.788168 · Cited by 8
Self-citationA Tool for the Accumulation and Evaluation of Multimodal Research
Coovert, M. D., Walvoord, A. A. G., Elliott, L. R., & Redden, E. S. (2008) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2008.2001695 · Cited by 8
Multimodal Input for Perceptual User Interfaces
LaViola, J. J., Buchanan, S., & Pittman, C. R. (2014) · doi:10.1002/9781118706237.ch9 · Cited by 8
Clustering Genes Using Heterogeneous Data Sources
Zeng, E., Yang, C., Li, T., & Narasimhan, G. (2010) · International Journal of Knowledge Discovery in Bioinformatics · doi:10.4018/jkdb.2010040102 · Cited by 7
Vision-Based Interaction within a Multimodal Framework
Sá, V. J., Malerczyk, C., & Schnaider, M. (2001) · Cited by 7
Web Multimedia Object Clustering via Information Fusion
Lu, W., Li, L., Li, T., Zhang, H., & Guo, J. H. (2011) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2011.72 · Cited by 6
Advances in the Robust Processing of Multimodal Speech and Pen Systems
Oviatt, S. L. (2002) · Series in machine perception and artificial intelligence · doi:10.1142/9789812778543_0008 · Cited by 6
Self-citationPfleger, N. (2007) · Publications of the UdS (Saarland University) · doi:10.22028/d291-25919 · Cited by 6
On-line sketch recognition for course of action diagrams
Wang, X., Xu, H., & Wang, H. (2010) · IEEE International Conference on Mechatronics and Automation (ICMA) · doi:10.1109/icma.2010.5588659 · Cited by 5
Max‐margin discriminative random fields for multimodal human action recognition
Su, Y., Ma, L., Liu, A., & Yang, Z. (2014) · Electronics Letters · doi:10.1049/el.2014.1027 · Cited by 5
Jiang, T., Luo, S., Wang, D., Li, Y., Wu, Y., Li, T., & Zhang, G. (2023) · Scientific Reports · doi:10.1038/s41598-023-37969-2 · Cited by 5
Machine Learning Approaches for Music Information Retrieval
Li, T., Ogihara, M., & Shao, B. (2009) · BiblioBoard Library Catalog (Open Research Library) · doi:10.5772/6687 · Cited by 5
Multimodal Interfaces to Mobile Terminals – A Design-For-All Approach
Kvale, K., & Dilp, N. (2010) · InTech eBooks · doi:10.5772/9499 · Cited by 5
A Natural Human-Computer Interface for Controlling Wheeled Robotic Vehicles
Flippo, F. (2003) · Cited by 5
Multimodal Architectures: Issues and Experiences
Frattini, G., Romano, L., di Carlo, V. S., Petriccione, P., Supino, G., Leone, G. G. M., & Autiero, C. (2006) · Lecture notes in computer science · doi:10.1007/11915034_122 · Cited by 5
Nakisa, B. (2019) · Queensland University of Technology · doi:10.5204/thesis.eprints.129875 · Cited by 5
Bimodal Combination Of Speech And Handwriting For Improved Word Recognition
Woodruff, P., & Dupont, S. (2005) · doi:10.5281/zenodo.39283 · Cited by 4
Modelling Stress Recognition in Conflict Resolution Scenarios
Gomes, M., Carneiro, D. R., Nováis, P., & Neves, J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-28942-2_48 · Cited by 4
Distributed Architecture for Intelligent Robotic Assembly Part I: Design and Multimodal Learning
López-Juárez, I., & Rios, R. (2006) · doi:10.5772/5054 · Cited by 4
Multimodal Human-Robot Interaction in an Assistive Technology Context
Harte, E., & Jarvis, R. A. (2008) · Cited by 4
Adaptive multimodal integration of speech and gaze
Mantravadi, C. S. (2009) · Rutgers University Community Repository (Rutgers University) · doi:10.7282/t3qc03pm · Cited by 4
MPNAS: Multimodal Sentiment Analysis Pruning via Neural Architecture Search
Zhang, B., Ren, A., Zhang, Z., Duan, M., Liu, D., Tan, Y., & Zhong, K. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887670 · Cited by 3
Shin, H.-J., Yoon, H., Kim, S., & Kang, D. (2022) · Applied Sciences · doi:10.3390/app12157355 · Cited by 3
Ambient Intelligence on Personal Mobility Assistants for Sustainable Travel Choices
Magliocchetti, D., Gielow, M., Devigili, F., Conti, G., & Amicis, R. D. (2012) · Journal of Ubiquitous Systems and Pervasive Networks · doi:10.5383/juspn.04.01.001 · Cited by 3
A multimedia information fusion framework for web image categorization
Lu, W., Li, L., Li, J., Li, T., Zhang, H., & Guo, J. H. (2012) · Multimedia Tools and Applications · doi:10.1007/s11042-012-1165-2 · Cited by 3
Graph-Based Partial Hypothesis Fusion for Pen-Aided Speech Input
Liu, P., & Soong, F. K. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2009.2013409 · Cited by 3
Towards a new approach for integrating multimodal user input based on evolutionary computation
Althoff, F., Al-Hames, M., McGlaun, G., & Lang, M. K. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745032 · Cited by 3
Exploring Interaction Between Images and Texts for Web Image Categorization
Li, L., Lu, W., Li, J., Li, T., Zhang, H., & Guo, J. H. (2011) · Cited by 3
Fotouhi, F. A., Reddy, C. K. A., & Alaydie, N. (2012) · Cited by 3
Peña, M., & López-Juárez, I. (2006) · doi:10.5772/5056 · Cited by 3
Learning to Recognize Hand-Held Objects from Scratch
Li, X., Jiang, S., Lv, X., & Chen, C. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-48896-7_52 · Cited by 3
Use of Data Mining in Emergency Evacuation Planning
Miah, M. (2012) · Cited by 3
Robot Vision Methodology for Assembly Manufacturing Tasks
Peña‐Cabrera, M., López-Juárez, I., & Osorio-Comparán, R. (2007) · Electronics, Robotics and Automotive Mechanics Conference · doi:10.1109/cerma.2007.4367701 · Cited by 3
Optimal decision fusion with applications to target detection in wireless ad hoc sensor networks
Duarte, M. F., & Hu, Y. H. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2004.1394606 · Cited by 3
Multimodal fusion for interaction systems
Wehbi, A., Zaguia, A., Ramdane-Chérif, A., & Tadj, C. (2013) · Cited by 2
Balasubramanian, N., & Rajasekar, E. (2023) · BENTHAM SCIENCE PUBLISHERS eBooks · doi:10.2174/9789815079210123010011 · Cited by 2
Multimodal Interfaces: A Study on Speech-Hand Gesture Recognition
Martinez, J. J. L., & Dewanti, S. S. (2019) · International Conference on Information and Communications Technology (ICOIACT) · doi:10.1109/icoiact46704.2019.8938421 · Cited by 2
Hui, P. Y., & Meng, H. M. L. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2011509 · Cited by 2
Cognitive Principles in Robust Multimodal Interpretation
Chai, J. Y., Prasov, Z., & Qu, S. (2006) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1936 · Cited by 2
Robot Vision Methodology for Assembly Manufacturing Tasks
López-Juárez, I., Lopez, J. I., & Osorio-Comparán, R. (2007) · Electronics, Robotics and Automotive Mechanics Conference · doi:10.1109/cerma.2007.108 · Cited by 2
Clustering Genes Using Heterogeneous Data Sources
Zeng, E., Yang, C., Li, T., & Narasimhan, G. (2013) · IGI Global eBooks · doi:10.4018/978-1-4666-1785-8.ch005 · Cited by 2
Optimization of Multimodal Japanese Teaching Model Using Virtual Reality
Ye, X. (2022) · Mobile Information Systems · doi:10.1155/2022/7364279 · Cited by 3
Use What You've Got: Steps Toward Opportunistic Computing
McGee, D. R., & Cohen, P. R. (2000) · doi:10.21236/ada420066 · Cited by 2
Self-citationNews-MESI: A Dataset for Multimodal News Excerpt Segmentation and Identification
Song, Q., Jia, Z., Jia, W., Zhao, W., Hu, M., & Liu, C. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3369866 · Cited by 1
User perception of multi-modal interfaces for mobile applications
Metze, F., Englert, R. A., Bub, U., Kliche, I., & Scheerbarth, T. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-612 · Cited by 1
Applications in Intelligent Speech Analysis
Schuller, B. W. (2013) · Signals and communication technology · doi:10.1007/978-3-642-36806-6_10 · Cited by 1
Multimodal Deixis: Nearby Reality or Still Science Fiction?
Ordelmans, L. (2008) · Cited by 1
Fusing Multimodal Human-Robot Communication using Deep Learning
Shukla, P., Bhandari, B., Goel, A., Charan, R., Gali, N., Dwivedi, A., Mundra, N., Agrawal, R., & Nandi, G. C. (2019) · IEEE Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON) · doi:10.1109/upcon47278.2019.8980213 · Cited by 1
Anticipatory and Invisible Interfaces to Address Impaired Proprioception in Neurological Disorders
Tadayon, A. (2020) · Cited by 1
"What's that called?": a multimodal fusion approach for cultural heritage virtual experiences.
Grazioso, M., Maro, M. D., & Cutugno, F. (2020) · Cited by 1
Usage patterns and latent semantic analyses for task goal inference of multimodal user interactions
Hui, P. Y., Lo, W.-K., & Meng, H. M. L. (2010) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1719970.1719989 · Cited by 1
Perlin, K., & Robbins, C. A. (2005) · Cited by 1
Visualizing Multimodal Interactions: Design and Evaluation of Experience Sharing
Boda, P., & Vetek, A. (2015) · National Conference on Artificial Intelligence
User-centric Music Information Retrieval
Shao, B. (2011) · doi:10.25148/etd.fi11041508
Visual Emotion Recognition: Status and Key Issues
Caplier, A. (2013) · Wiley eBooks · doi:10.1002/9781118601938.ch4
Dual-sensor fusion for seamless indoor user localisation and tracking
Redzic, M. (2013) · Arrow@dit (Dublin Institute of Technology)
A Realization of Auto Plotting Table Basing on MMHCI
Jin, J., & Li, W. (2010) · Machine Vision and Human-machine Interface · doi:10.1109/mvhi.2010.206
Combining Classical and Embodied Multimodal Fusion for Human-Robot Interaction
Giuliani, M., & de Ruiter, J. (2011) · mediaTUM (Technical University of Munich)
Mutual information as a variable to differentiate the roles of gaze in the multimodal interface
Cooke, N. J., & Shen, A. (2010) · Workshop on Eye Gaze in Intelligent Human Machine Interaction (GazeIn) · doi:10.1145/2002333.2002342
Real Time Object Recognition Methodology
Cabrera, M. P., López-Juárez, I., Cabrera, R. R., Osorio-Comparán, R., & Gomez, H. (2010) · Electronics, Robotics and Automotive Mechanics Conference · doi:10.1109/cerma.2010.116
Feedback Information Expression and Fusion Method for Human-Robot Interaction
Zhao, Q., Tu, D., Lü, J., & Huang, Z. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-87442-3_109
Collaborative multimodal authoring of virtual worlds
Sá, V. J., Marreiros, F. M. M., & Marcos, A. (2005) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
A semantic fusion based Multimodal Interaction Panel for virtual city planning
He, Y., Geng, G., Zhou, M., Li, K., Du, Z., & Zhao, Y. (2011) · IEEE International Conference on Computer Science and Automation Engineering · doi:10.1109/csae.2011.5952877
A Survey of the Information Fusion in MMHCI
Jin, J., & Li, W. (2010) · Machine Vision and Human-machine Interface · doi:10.1109/mvhi.2010.207
Applications in Intelligent Speech Analysis
Fitzgerald, F. S. (2013)
AnAgent-Based Multimodal Interface forSketchInterpretation
Azar, S., Couvreurt, L., Delfosse, V., Jaspart, B., & Polytechnique, F. (2006)
User interfaces for anyone anywhere
Sá, V. J. (2001)
Multimodal Integration in Embodied Conversational Kiosk with Augmented Reality
Huang, J. H. (2001) · DSpace@MIT (Massachusetts Institute of Technology)
Distributed architecture for intelligent robotic agents for assembly
Rìos-Cabrera, R., & López-Juárez, I. (2006) · International Conference on Systems
Avola, D., Buono, A. D., & Spognardi, A. (2011) · Advances in human and social aspects of technology book series · doi:10.4018/978-1-61692-904-6.ch011
Augmenting Physical Tools: Bridging the Chasm of Late Adoption with Multimodal Language
McGee, D. R., Cohen, P. R., & Smith, G. S. M. J. (2001)
Self-citationDealing with multimodal languages ambiguities : a classification and solution method
Caschera, M. C. (2009)
Kamikawa, K., Maeda, K., Ogawa, T., & Haseyama, M. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222279
Context-Aware Embodied Question Answering Agent With Large Language Model
Wang, Z., Luo, H., Jiang, B., Li, J., Zheng, J., & Bai, X. P. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3668526
Multimodal Fusion and Adaptive Learning for Cardiopulmonary Classification
Abed, M. T., & Zhao, X. (2026) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3807503.3819467
Karim, T. A. (2007) · RIT Scholar Works (Rochester Institute of Technology)
Metadata
- Authors
- Wu, L., Oviatt, S. L., & Cohen, P. R.
- Venue
- IEEE Transactions on Multimedia
- Published
- 1999
- Volume
- 1
- Issue
- 4
- Pages
- 334-341
- ISSN
- 1520-9210 (ISSN-L)
- OpenAlex
- W2151194828
- Cited by
- 190
- References
- 34
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 2 · 2023: 5 · 2022: 4 · 2021: 1 · 2020: 5 · 2019: 5 · 2018: 4 · 2017: 7 · 2016: 3 · 2015: 3 · 2014: 5 · 2013: 8 · 2012: 8
- FWCI
- 9.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- multimodal recognition, multimodal integration, robustness, semantic fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review