Multimodal integration-a statistical view
Abstract
We present a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We first identify the primary factors that influence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estimate approach and a learning approach, which are designed to optimize accurate recognition during the multimodal integration process. We evaluate these methods using Quickset, a speech/gesture multimodal system, and report evaluation results based on an empirical corpus collected with Quickset. From an architectural perspective, the integration technique presented offers enhanced robustness. It also is premised on more realistic assumptions than previous multimodal systems using semantic fusion. From a methodological standpoint, the evaluation techniques that we describe provide a valuable tool for evaluating multimodal systems.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Masked Language Modelling in the text domain has been applied to audio and image, e.g., Masked Acoustic Modelling and Masked Image Region Prediction.
“For instance, Masked Language Modelling (MLM) in the text domain has been applied to audio and image, e.g., Masked Acoustic Modelling [180], [200], Masked Image Region Prediction [190]”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [200]
Cites
Pattern classification and scene analysis
Grenander, U., Duda, R., & Hart, P. (1974) · A Wiley-Interscience publication · doi:10.2307/1573081 · Cited by 17,399 · see citing works
Pattern classification and scene analysis
Duda, R. O., & Hart, P. E. (1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895 · see citing works
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,312 · see citing works
Bolt, R. A. (1980) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/800250.807503 · Cited by 1,445 · see citing works
Bridle, J. S. (1990) · Springer eBooks · doi:10.1007/978-3-642-76153-9_28 · Cited by 1,278 · see citing works
Bolt, R. A. (1980) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965105.807503 · Cited by 921 · see citing works
The logic of typed feature structures
Carpenter, B. (1992) · Cited by 600 · see citing works
Cohen, P. R., Johnston, M. O., McGee, D. R., Oviatt, S. L., Pittman, J., Smith, I., Chen, L., & Clow, J. (1997) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/266180.266328 · Cited by 466 · see citing works
Self-citationData fusion in robotics and machine intelligence
Abidi, M. A., & Gonzalez, R. C. (1992) · Academic Press Professional, Inc. eBooks · Cited by 444 · see citing works
The Logic of Typed Feature Structures
Carpenter, R. L. (1992) · Cambridge University Press eBooks · doi:10.1017/cbo9780511530098 · Cited by 410 · see citing works
A word graph algorithm for large vocabulary continuous speech recognition
Ortmanns, S., Ney, H., & Aubert, X. L. (1997) · Computer Speech & Language · doi:10.1006/csla.1996.0022 · Cited by 400 · see citing works
Integration and synchronization of input modes during multimodal human-computer interaction
Oviatt, S. L., Angeli, A. D., & Kuhn, K. (1997) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/258549.258821 · Cited by 368 · see citing works
Self-citationMutual disambiguation of recognition errors in a multimodel architecture
Oviatt, S. L. (1999) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/302979.303163 · Cited by 317 · see citing works
Self-citationToward multimodal human-computer interface
Sharma, R., Pavlović, V., & Huang, T. S. (1998) · Proceedings of the IEEE · doi:10.1109/5.664275 · Cited by 303 · see citing works
Data fusion in robotics and machine intelligence
Prasad, K. V. (1993) · Control Engineering Practice · doi:10.1016/0967-0661(93)91608-y · Cited by 247 · see citing works
An Overview of Predictive Learning and Function Approximation
Friedman, J. H. (1994) · Springer eBooks · doi:10.1007/978-3-642-79119-2_1 · Cited by 190 · see citing works
Unification-based multimodal integration
Johnston, M. O., Cohen, P. R., McGee, D. R., Oviatt, S. L., Pittman, J. A., & Smith, I. (1997) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/976909.979653 · Cited by 177 · see citing works
Self-citationMultimodal interfaces for dynamic interactive maps
Oviatt, S. L. (1996) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/238386.238438 · Cited by 175 · see citing works
Self-citationSynergistic use of direct manipulation and natural language
Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C. N., & Sullivan, J. (1989) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/67449.67494 · Cited by 143 · see citing works
Self-citationIntelligent multi-media interface technology
Neal, J. G., & Shapiro, S. C. (1991) · Intelligent User Interfaces · doi:10.1145/107215.128690 · Cited by 112 · see citing works
Vo, M. T., & Wood, C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.550794 · Cited by 80 · see citing works
Adaptive bimodal sensor fusion for automatic speechreading
Meier, U., Hürst, W., & Duchnowski, P. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543250 · Cited by 77 · see citing works
Interactive simulation in a multi-person virtual world
Codella, C. F., Sweeney, P., Turk, G., Jalili, R., Koved, L., Lewis, J. B., Ling, D. T., Lipscomb, J. S., Rabenhorst, D. A., Wang, C. P., et al. (1992) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/142750.142825 · Cited by 72 · see citing works
Waibel, A., Vo, M. T., Duchnowski, P., & Manke, S. (1996) · Artificial Intelligence Review · doi:10.1007/bf00127684 · Cited by 69 · see citing works
“Finger-Pointer”: Pointing interface by image processing
Fukumoto, M., Suenaga, Y., & Mase, K. (1994) · Computers & Graphics · doi:10.1016/0097-8493(94)90157-0 · Cited by 65 · see citing works
Integrating simultaneous input from speech, gaze, and hand gestures
Koons, D. B., Sparrell, C. J., & Þórisson, K. R. (1991) · National Conference on Artificial Intelligence · Cited by 47 · see citing works
Speech/gesture interface to a visual computing environment for molecular biologists
Sharma, R., Huang, T. S., Pavlović, V., Zhao, Y., Lo, Z., Chu, S., & Schul, K. (1996) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1996.547311 · Cited by 45 · see citing works
Yaeger, L., Webb, B. J., & Lyon, R. F. (1998) · AI Magazine · doi:10.1609/aimag.v19i1.1355 · Cited by 35 · see citing works
Integration of eye-gaze, voice and manual response in multimodal user interface
Wang, J. (2002) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1995.538404 · Cited by 35 · see citing works
Yaeger, L., Webb, B. J., & Lyon, R. F. (1998) · Lecture notes in computer science · doi:10.1007/3-540-49430-8_14 · Cited by 20 · see citing works
Combining Visual and Acoustic Speech Signals with a Neural Network Improves Intelligibility
B, T. J. S. A., Yuhas, B. P., Goldstein, M. H., & Jenkins, R. E. (1989) · neural information processing systems · Cited by 17 · see citing works
Flagged first, then by citations
3 of the 34 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
Schuller, B. W., Batliner, A. M., Steidl, S., & Seppi, D. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.011 · Cited by 765 · see citing works
Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning
Ong, S. C. W., & Ranganath, S. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.112 · Cited by 562 · see citing works
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Multimodal interaction: A review
Turk, M. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2013.07.003 · Cited by 519 · see citing works
Oviatt, S. L. (2007) · Human factors and ergonomics · doi:10.1201/9781410615862.ch21 · Cited by 485 · see citing works
Self-citationAutomatic, Dimensional and Continuous Emotion Recognition
Gunes, H., & Pantic, M. (2010) · International Journal of Synthetic Emotions · doi:10.4018/jse.2010101605 · Cited by 466 · see citing works
Bi-modal emotion recognition from expressive face and body gestures
Gunes, H., & Piccardi, M. (2007) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2006.09.007 · Cited by 382 · see citing works
Oviatt, S. L., Cohen, P. R., Wu, L., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T. A., Landay, J. A., Larson, J., et al. (2000) · Human-Computer Interaction · doi:10.1207/s15327051hci1504_1 · Cited by 377 · see citing works
Self-citationMultimodal Interfaces: A Survey of Principles, Models and Frameworks
Dumas, B., Lalanne, D., & Oviatt, S. L. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-00437-7_1 · Cited by 308 · see citing works
Self-citationTaming recognition errors with a multimodal interface
Oviatt, S. L. (2000) · Communications of the ACM · doi:10.1145/348941.348979 · Cited by 300 · see citing works
Self-citationGesture interaction in virtual reality
Li, Y., Huang, J., Tian, F., Wang, H., & Dai, G. (2019) · Virtual Reality & Intelligent Hardware · doi:10.3724/sp.j.2096-5796.2018.0006 · Cited by 279 · see citing works
Perceptual user interfaces: multimodal interfaces that process what comes naturally
Oviatt, S. L., & Cohen, P. R. (2000) · Communications of the ACM · doi:10.1145/330534.330538 · Cited by 277 · see citing works
Self-citationSurvey of data management and analysis in disaster situations
Hristidis, V., Chen, S., Li, T., Luis, S., & Deng, Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.04.065 · Cited by 225 · see citing works
A review of speech-based bimodal recognition
Chibelushi, C. C., Deravi, F., & Mason, J. S. (2002) · IEEE Transactions on Multimedia · doi:10.1109/6046.985551 · Cited by 222 · see citing works
Automatic Temporal Segment Detection and Affect Recognition From Face and Body Display
Gunes, H., & Piccardi, M. (2009) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.927269 · Cited by 225 · see citing works
MULTIMODAL INTERFACES THAT PROCESS WHAT COMES NATURALLY
Oviatt, S. L., & Cohen, P. R. (2000) · Cited by 196 · see citing works
Self-citationSchuller, B. W., Müller, R., Eyben, F., Gast, J., Hörnler, B., Wöllmer, M., Rigoll, G., Hoethker, A., & Konosu, H. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.02.013 · Cited by 194 · see citing works
Affect Recognition from Face and Body: Early Fusion vs. Late Fusion
Gunes, H., & Piccardi, M. (2006) · 2005 IEEE International Conference on Systems, Man and Cybernetics · doi:10.1109/icsmc.2005.1571679 · Cited by 191 · see citing works
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality
Kaiser, E., Olwal, A., McGee, D. R., Benko, H., Corradini, A., Li, X., Cohen, P. R., & Feiner, S. (2003) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/958432.958438 · Cited by 179 · see citing works
Self-citationAlvarado, C., & Davis, R. (2004) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1029632.1029637 · Cited by 158 · see citing works
Li, T., & Ding, C. H. Q. (2008) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972788.72 · Cited by 143 · see citing works
Orko: Facilitating Multimodal Interaction for Visual Exploration and Analysis of Networks
Srinivasan, A., & Stasko, J. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2745219 · Cited by 142 · see citing works
Generation and evaluation of user tailored responses in multimodal dialogue
Walker, M. A., Whittaker, S. J., Stent, A., Maloor, P., Moore, J., Johnston, M., & Vasireddy, G. (2004) · Cognitive Science · doi:10.1016/j.cogsci.2004.06.002 · Cited by 125 · see citing works
Understanding vision-based continuous sign language recognition
Aloysius, N., & Geetha, M. K. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-08961-z · Cited by 122 · see citing works
Handbook of Research on User Interface Design and Evaluation for Mobile Technology
Lumsden, J. (2008) · IGI Global eBooks · doi:10.4018/978-1-59904-871-0 · Cited by 118 · see citing works
Finite-state multimodal parsing and understanding
Johnston, M. O., & Bangalore, S. (2000) · International Conference on Computational Linguistics · doi:10.3115/990820.990874 · Cited by 116 · see citing works
Advances in robust multimodal interface design
Oviatt, S. L. (2003) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2003.1231179 · Cited by 108 · see citing works
Self-citationUnobtrusive sleep state measurements in preterm infants – A review
Werth, J., Atallah, L., Andriessen, P., Long, X., Zwartkruis-Pelgrim, E., & Aarts, R. M. (2016) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2016.03.005 · Cited by 106 · see citing works
Automatic Emotion Recognition Using Temporal Multimodal Deep Learning
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020) · IEEE Access · doi:10.1109/access.2020.3027026 · Cited by 101 · see citing works
Predicting MCI Status From Multimodal Language Data Using Cascaded Classifiers
Fraser, K., Fors, K. L., Eckerström, M., Öhman, F., & Kokkinakis, D. (2019) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2019.00205 · Cited by 93 · see citing works
From the Lab to the Real World: Affect Recognition Using Multiple Cues and Modalities
Güneş, H., Piccardi, M., & Pantić, M. (2008) · doi:10.5772/6180 · Cited by 93 · see citing works
On probabilistic combination of face and gait cues for identification
Shakhnarovich, G., & Darrell, T. (2003) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2002.1004151 · Cited by 90 · see citing works
Holzapfel, H., Nickel, K., & Stiefelhagen, R. (2004) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1027933.1027964 · Cited by 88 · see citing works
Coupled hidden conditional random fields for RGB-D human action recognition
Liu, A.-A., Nie, W., Su, Y., Ma, L., Hao, T., & Yang, Z. (2014) · Signal Processing · doi:10.1016/j.sigpro.2014.08.038 · Cited by 87 · see citing works
Wöllmer, M., Al-Hames, M., Eyben, F., Schuller, B. W., & Rigoll, G. (2009) · Neurocomputing · doi:10.1016/j.neucom.2009.08.005 · Cited by 86 · see citing works
Multi-view latent variable discriminative models for action recognition
Song, Y., Morency, L., & Davis, R. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247918 · Cited by 81 · see citing works
Recognizing Affect from Linguistic Information in 3D Continuous Space
Schuller, B. W. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.17 · Cited by 69 · see citing works
Qu, X., Liu, Z., Wu, C. Q., Hou, A., Yin, X., & Chen, Z. (2024) · Sensors · doi:10.3390/s24020637 · Cited by 63 · see citing works
Breaking the Robustness Barrier: Recent Progress on the Design of Robust Multimodal Systems
Oviatt, S. L. (2002) · Advances in computers · doi:10.1016/s0065-2458(02)80009-2 · Cited by 62 · see citing works
Self-citationAlvarado, C., & Davis, R. (2007) · ACM SIGGRAPH Courses · doi:10.1145/1281500.1281545 · Cited by 59 · see citing works
Image-text dual neural network with decision strategy for small-sample image classification
Zhu, F., Ma, Z., Li, X., Chen, G., Chien, J., Xue, J., & Guo, J. H. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.02.099 · Cited by 58 · see citing works
DNN Multimodal Fusion Techniques for Predicting Video Sentiment
Williams, J. A., Comanescu, R., Radu, O., & Tian, L. (2018) · doi:10.18653/v1/w18-3309 · Cited by 56 · see citing works
Multimodality Sensor System for Long-Term Sleep Quality Monitoring
Peng, Y.-T., Lin, C., Sun, M., & Landis, C. A. (2007) · IEEE Transactions on Biomedical Circuits and Systems · doi:10.1109/tbcas.2007.914481 · Cited by 56 · see citing works
Machine vision approach for robotic assembly
Peña‐Cabrera, M., López-Juárez, I., Rìos-Cabrera, R., & Corona‐Castuera, J. (2005) · Assembly Automation · doi:10.1108/01445150510610926 · Cited by 53 · see citing works
Creating tangible interfaces by augmenting physical objects with multimodal language
McGee, D. R., & Cohen, P. R. (2001) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/359784.360305 · Cited by 53 · see citing works
Self-citationUser and context adaptive neural networks for emotion recognition
Caridakis, G., Karpouzis, K. C., & Kollias, S. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.11.043 · Cited by 51 · see citing works
McGee, D. R., Cohen, P. R., & Wu, L. (2000) · Designing Augmented Reality Environments (DARE) · doi:10.1145/354666.354674 · Cited by 50 · see citing works
Self-citationSun, B., Cao, S., He, J. Y. D., & Yu, L. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.11.021 · Cited by 46 · see citing works
Corradini, A., & Cohen, P. R. (2003) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2002.1007499 · Cited by 45 · see citing works
Self-citationDynamically constructed Bayes nets for multi-domain sketch understanding
Alvarado, C., & Davis, R. (2007) · ACM SIGGRAPH Courses · doi:10.1145/1281500.1281544 · Cited by 44 · see citing works
The Paradigm Shift to Multimodality in Contemporary Computer Interfaces
Oviatt, S. L., & Cohen, P. R. (2015) · Synthesis lectures on human-centered informatics · doi:10.2200/s00636ed1v01y201503hci030 · Cited by 58 · see citing works
Self-citationMultimodal user’s affective state analysis in naturalistic interaction
Caridakis, G., Karpouzis, K. C., Wallace, M., Kessous, L., & Amir, N. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0030-8 · Cited by 38 · see citing works
Semisupervised learning from different information sources
Li, T., & Ogihara, M. (2004) · Knowledge and Information Systems · doi:10.1007/s10115-004-0155-8 · Cited by 35 · see citing works
Multi-view Generative Adversarial Networks
Chen, M., & Denoyer, L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71246-8_11 · Cited by 33 · see citing works
EMBASSI multimodal assistance for infotainment and service infrastructures
Herfet, T., Kirste, T., & Schnaider, M. (2001) · Computers & Graphics · doi:10.1016/s0097-8493(01)00086-3 · Cited by 33 · see citing works
Güneş, H., & Piccardi, M. (2005) · Lecture notes in computer science · doi:10.1007/11573548_14 · Cited by 32 · see citing works
Recognition of Emotional States in Natural Human-Computer Interaction
Cowie, R., Douglas‐Cowie, E., Karpouzis, K. C., Caridakis, G., Wallace, M., & Kollias, S. (2008) · Springer eBooks · doi:10.1007/978-3-540-78345-9_6 · Cited by 33 · see citing works
Liu, J., & Kavakli, M. (2010) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2010.5583252 · Cited by 32 · see citing works
From members to teams to committee-a robust approach to gestural and multimodal recognition
Wu, L., Oviatt, S. L., & Cohen, P. R. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1021897 · Cited by 30 · see citing works
Self-citationMultimodal Integration - A Biological View
Coen, M. H. (2001) · Cited by 29 · see citing works
Multimodal interface research: a science without borders
Oviatt, S. L. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-464 · Cited by 28 · see citing works
Self-citationFusing face and body gesture for machine recognition of emotions
Gunes, H., & Piccardi, M. (2006) · IEEE RO-MAN · doi:10.1109/roman.2005.1513796 · Cited by 28 · see citing works
Multi-domain sketch understanding
Alvarado, C. (2007) · ACM SIGGRAPH Courses · doi:10.1145/1281500.1281518 · Cited by 27 · see citing works
A Salience-Based Approach to Gesture-Speech Alignment.
Eisenstein, J., & Christoudias, C. M. (2004) · Cited by 26 · see citing works
Emotion Recognition in Naturalistic Speech and Language—A Survey
Weninger, F. J., Wöllmer, M., & Schuller, B. W. (2015) · Wiley eBooks · doi:10.1002/9781118910566.ch10 · Cited by 26 · see citing works
Learning and Knowledge-Based Sentiment Analysis in Movie Review Key Excerpts
Schuller, B. W., & Knaup, T. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-18184-9_39 · Cited by 25 · see citing works
Dynamically constructed Bayes nets for multi-domain sketch understanding
Alvarado, C., & Davis, R. (2006) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1185657.1185793 · Cited by 25 · see citing works
Optimization in multimodal interpretation
Chai, J. Y., Hong, P., Zhou, M. X., & Prasov, Z. (2004) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1218955.1218956 · Cited by 25 · see citing works
EMBASSI: Electronic Multimedia and Service Assistance
Hildebrand, A., & Sá, V. J. (2022) · doi:10.24406/publica-fhg-335797 · Cited by 21 · see citing works
WavoID: Robust and Secure Multi-modal User Identification via mmWave-voice Mechanism
Liu, T., Lin, F., Wang, C., Xu, C., Zhang, X., Li, Z., Xu, W., Huang, M.-C., & Ren, K. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606775 · Cited by 20 · see citing works
Developing an Ambient Intelligent-Based Decision Support System for Production and Control Planning
Gomes, M., Silva, F., Ferraz, F., Silva, A., Analide, C., & Nováis, P. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-53480-0_97 · Cited by 20 · see citing works
Miki, M., Kitaoka, N., Miyajima, C., Nishino, T., & Takeda, K. (2014) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/1687-4722-2014-2 · Cited by 18 · see citing works
Gene functional classification by semi-supervised learning from heterogeneous data
Li, T., Shenghuo Zhu, B., Li, Q., & Ogihara, M. (2003) · ACM Symposium on Applied Computing · doi:10.1145/952532.952552 · Cited by 18 · see citing works
Web Multimedia Object Classification Using Cross-Domain Correlation Knowledge
Lu, W., Li, J., Li, T., Guo, W., Zhang, H., & Guo, J. H. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2280895 · Cited by 17 · see citing works
An Agent-Based Multimodal Interface for Sketch Interpretation
Azar, S., Couvreur, L., Delfosse, V., Jaspart, B., & Boulanger, C. (2006) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2006.285357 · Cited by 17 · see citing works
Cognitive principles in robust multimodal interpretation
Chai, J. Y., Prasov, Z., & Qu, S. (2008) · Cited by 17 · see citing works
Latent Semantic Analysis for Multimodal User Input With Speech and Gestures
Hui, P. Y., & Meng, H. M. L. (2014) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2013.2294586 · Cited by 16 · see citing works
Automatic, Dimensional and Continuous Emotion Recognition
Güneş, H., & Pantić, M. (2012) · IGI Global eBooks · doi:10.4018/978-1-4666-1595-3.ch005 · Cited by 16 · see citing works
Multimodal Human-Computer Interaction
Turk, M. (2005) · Springer eBooks · doi:10.1007/0-387-27890-7_16 · Cited by 16 · see citing works
Machine learning in low-level microarray analysis
Rubinstein, B. I. P., McAuliffe, J., Cawley, S. E., Palaniswami, M. S., Ramamohanarao, K., & Speed, T. P. (2003) · ACM SIGKDD Explorations Newsletter · doi:10.1145/980972.980988 · Cited by 16 · see citing works
Ruiz, N., Chen, F., & Oviatt, S. L. (2010) · Elsevier eBooks · doi:10.1016/b978-0-12-374825-6.00010-1 · Cited by 15 · see citing works
Self-citationIntegrating Features from Different Sources for Music Information Retrieval
Li, T., Ogihara, M., & Shenghuo Zhu, B. (2006) · Proceedings · doi:10.1109/icdm.2006.89 · Cited by 15 · see citing works
Kirste, T., Herfet, T., & Schnaider, M. (2001) · EC/NSF Workshop on Universal Accessibility of Ubiquitous Computing: Providing for the Elderly (WUAUC) · doi:10.1145/564526.564539 · Cited by 15 · see citing works
Music Clustering With Features From Different Information Sources
Li, T., Ogihara, M., Peng, W., Shao, B., & Shenghuo Zhu, B. (2009) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2009.2012942 · Cited by 14 · see citing works
Multimodal Fusion of Smart Home and Text-based Behavior Markers for Clinical Assessment Prediction
Sprint, G., Cook, D. J., Schmitter‐Edgecombe, M., & Holder, L. B. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3531231 · Cited by 13 · see citing works
Natural interaction with a virtual guide in a virtual environment
Hofs, D. H. W., Theune, M., & den Akker, R. O. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-009-0024-6 · Cited by 13 · see citing works
Joint interpretation of input speech and pen gestures for multimodal human-computer interaction
Hui, P. Y., & Meng, H. M. L. (2006) · Interspeech (USB) · doi:10.21437/interspeech.2006-362 · Cited by 13 · see citing works
Mind: A Context-Based Multimodal Interpretation Framework in Conversational Systems
Chai, J. Y., Pan, S., & Zhou, M. X. (2005) · Text, speech and language technology · doi:10.1007/1-4020-3933-6_12 · Cited by 13 · see citing works
A visual modality for the augmentation of paper
McGee, D. R., Misha Pavel, C., Adami, A. M., Wang, G., & Cohen, P. R. (2001) · Workshop on Perceptive User Interfaces · doi:10.1145/971478.971480 · Cited by 13 · see citing works
Self-citationGaze-contingent automatic speech recognition
Cooke, N. J., & Russell, M. J. (2008) · IET Signal Processing · doi:10.1049/iet-spr:20070127 · Cited by 12 · see citing works
A Personal Mobility Assistant based on Ambient Intelligence to Promote Sustainable Travel Choices
Magliocchetti, D., Gielow, M., Vigili, F. D., Conti, G., & Amicis, R. D. (2011) · Procedia Computer Science · doi:10.1016/j.procs.2011.07.124 · Cited by 12 · see citing works
Paulo, J., Asvadi, A., Peixoto, P., & Amorim, P. (2017) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2017.08.002 · Cited by 12 · see citing works
Schuller, B. W., Metze, F., Steidl, S., Batliner, A. M., Eyben, F., & Polzehl, T. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5494986 · Cited by 12 · see citing works
Multimodal Integration (Image and Text) Using Ontology Alignment
Abu-Shareha, A. A., Rajeswari, M., & Ramachandram, D. (2009) · American Journal of Applied Sciences · doi:10.3844/ajassp.2009.1217.1224 · Cited by 12 · see citing works
Context Shifts: Extending the Meanings of Physical Objects With Language
McGee, D. R., Misha Pavel, C., & Cohen, P. R. (2001) · Human-Computer Interaction · doi:10.1207/s15327051hci16234_15 · Cited by 11 · see citing works
Self-citationSemi-Synchronous Speech and Pen Input
Watanabe, Y., Iwata, K., Nakagawa, R., Shinoda, K., & Furui, S. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366936 · Cited by 11 · see citing works
Human Emotion Recognition from Body Language of the Head using Soft Computing Techniques
Zhao, Y. (2012) · uO Research (University of Ottawa) · doi:10.20381/ruor-6160 · Cited by 11 · see citing works
Clustering genes using gene expression and text literature data
Yang, C., Zeng, E., Li, T., & Narasimhan, G. (2005) · IEEE Computational Systems Bioinformatics Conference (CSB) · doi:10.1109/csb.2005.23 · Cited by 10 · see citing works
- Authors
- Wu, L., Oviatt, S. L., & Cohen, P. R.
- Venue
- IEEE Transactions on Multimedia
- Published
- 1999
- Volume
- 1
- Issue
- 4
- Pages
- 334-341
- ISSN
- 1520-9210 (ISSN-L)
- OpenAlex
- W2151194828
- Cited by
- 190
- References
- 34
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 2 · 2023: 5 · 2022: 4 · 2021: 1 · 2020: 5 · 2019: 5 · 2018: 4 · 2017: 7 · 2016: 3 · 2015: 3 · 2014: 5 · 2013: 8 · 2012: 8
- FWCI
- 9.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- multimodal recognition, multimodal integration, robustness, semantic fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
