The DOI matches its Crossref record; 31 of the 34 references it cites were found in a registry or an index.
Multimodal integration-a statistical view
Abstract
We present a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We first identify the primary factors that influence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estimate approach and a learning approach, which are designed to optimize accurate recognition during the multimodal integration process. We evaluate these methods using Quickset, a speech/gesture multimodal system, and report evaluation results based on an empirical corpus collected with Quickset. From an architectural perspective, the integration technique presented offers enhanced robustness. It also is premised on more realistic assumptions than previous multimodal systems using semantic fusion. From a methodological standpoint, the evaluation techniques that we describe provide a valuable tool for evaluating multimodal systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For instance, Masked Language Modelling (MLM) in the text domain has been applied to audio and image, e.g., Masked Acoustic Modelling [180], [200], Masked Image Region Prediction [190]”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [200]
Citations
Cites
Pattern classification and scene analysis
Grenander, U., Duda, R., & Hart, P. (1974) · A Wiley-Interscience publication · doi:10.2307/1573081 · Cited by 17,399
(1973) · CERN Document Server (European Organization for Nuclear Research) · Cited by 12,895
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,394
Bolt, R. A. (1980) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/800250.807503 · Cited by 1,493
Bridle, J. S. (1990) · Springer eBooks · doi:10.1007/978-3-642-76153-9_28 · Cited by 1,278
Bolt, R. A. (1980) · ACM SIGGRAPH Computer Graphics · doi:10.1145/965105.807503 · Cited by 921
The logic of typed feature structures
Carpenter, B. (1992) · Cited by 600
Cohen, P. R., Johnston, M. O., McGee, D. R., Oviatt, S. L., Pittman, J., Smith, I., Chen, L., & Clow, J. (1997) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/266180.266328 · Cited by 466
Self-citationData fusion in robotics and machine intelligence
Abidi, M. A., & Gonzalez, R. C. (1992) · Academic Press Professional, Inc. eBooks · Cited by 444
The Logic of Typed Feature Structures
Carpenter, R. L. (1992) · Cambridge University Press eBooks · doi:10.1017/cbo9780511530098 · Cited by 410
A word graph algorithm for large vocabulary continuous speech recognition
Ortmanns, S., Ney, H., & Aubert, X. L. (1997) · Computer Speech & Language · doi:10.1006/csla.1996.0022 · Cited by 400
Integration and synchronization of input modes during multimodal human-computer interaction
Oviatt, S. L., Angeli, A. D., & Kuhn, K. (1997) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/258549.258821 · Cited by 368
Self-citationMutual disambiguation of recognition errors in a multimodel architecture
Oviatt, S. L. (1999) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/302979.303163 · Cited by 317
Self-citationToward multimodal human-computer interface
Sharma, R., Pavlović, V., & Huang, T. S. (1998) · Proceedings of the IEEE · doi:10.1109/5.664275 · Cited by 303
Data fusion in robotics and machine intelligence
Prasad, K. V. (1993) · Control Engineering Practice · doi:10.1016/0967-0661(93)91608-y · Cited by 247
An Overview of Predictive Learning and Function Approximation
Friedman, J. H. (1994) · Springer eBooks · doi:10.1007/978-3-642-79119-2_1 · Cited by 191
Unification-based multimodal integration
Johnston, M. O., Cohen, P. R., McGee, D. R., Oviatt, S. L., Pittman, J. A., & Smith, I. (1997) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/976909.979653 · Cited by 177
Self-citationMultimodal interfaces for dynamic interactive maps
Oviatt, S. L. (1996) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/238386.238438 · Cited by 175
Self-citationSynergistic use of direct manipulation and natural language
Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C. N., & Sullivan, J. (1989) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/67449.67494 · Cited by 143
Self-citationIntelligent multi-media interface technology
Neal, J. G., & Shapiro, S. C. (1991) · Intelligent User Interfaces · doi:10.1145/107215.128690 · Cited by 112
Vo, M. T., & Wood, C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.550794 · Cited by 80
Adaptive bimodal sensor fusion for automatic speechreading
Meier, U., Hürst, W., & Duchnowski, P. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1996.543250 · Cited by 77
Interactive simulation in a multi-person virtual world
Codella, C. F., Sweeney, P., Turk, G., Jalili, R., Koved, L., Lewis, J. B., Ling, D. T., Lipscomb, J. S., Rabenhorst, D. A., Wang, C. P., et al. (1992) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/142750.142825 · Cited by 72
Waibel, A., Vo, M. T., Duchnowski, P., & Manke, S. (1996) · Artificial Intelligence Review · doi:10.1007/bf00127684 · Cited by 69
“Finger-Pointer”: Pointing interface by image processing
Fukumoto, M., Suenaga, Y., & Mase, K. (1994) · Computers & Graphics · doi:10.1016/0097-8493(94)90157-0 · Cited by 65
Integrating simultaneous input from speech, gaze, and hand gestures
Koons, D. B., Sparrell, C. J., & Þórisson, K. R. (1991) · National Conference on Artificial Intelligence · Cited by 47
Speech/gesture interface to a visual computing environment for molecular biologists
Sharma, R., Huang, T. S., Pavlović, V., Zhao, Y., Lo, Z., Chu, S., & Schul, K. (1996) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1996.547311 · Cited by 45
Yaeger, L., Webb, B. J., & Lyon, R. F. (1998) · AI Magazine · doi:10.1609/aimag.v19i1.1355 · Cited by 35
Integration of eye-gaze, voice and manual response in multimodal user interface
Wang, J. (2002) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1995.538404 · Cited by 35
Yaeger, L., Webb, B. J., & Lyon, R. F. (1998) · Lecture notes in computer science · doi:10.1007/3-540-49430-8_14 · Cited by 20
Combining Visual and Acoustic Speech Signals with a Neural Network Improves Intelligibility
B, T. J. S. A., Yuhas, B. P., Goldstein, M. H., & Jenkins, R. E. (1989) · neural information processing systems · Cited by 17
Research works printed but not linked: 3
- poddar, 1998, toward natural gesture/speech hci: a case study of weather narration, Proc 1998 Workshop on Perceptual User Interfaces (PUI 98), 1
- wu, 0, From members to teams to committee— A robust approach to gestural and multimodal recognition
- young, 1996, Large vocabulary continuous speech recognition A review
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Schuller, B. W., Batliner, A. M., Steidl, S., & Seppi, D. (2011) · Speech Communication · doi:10.1016/j.specom.2011.01.011 · Cited by 765
Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning
Ong, S. C. W., & Ranganath, S. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.112 · Cited by 562
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527
Multimodal interaction: A review
Turk, M. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2013.07.003 · Cited by 554
Metadata
- Authors
- Wu, L., Oviatt, S. L., & Cohen, P. R.
- Venue
- IEEE Transactions on Multimedia
- Published
- 1999
- Volume
- 1
- Issue
- 4
- Pages
- 334-341
- ISSN
- 1520-9210 (ISSN-L)
- OpenAlex
- W2151194828
- Cited by
- 190
- References
- 34
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 2 · 2023: 5 · 2022: 4 · 2021: 1 · 2020: 5 · 2019: 5 · 2018: 4 · 2017: 7 · 2016: 3 · 2015: 3 · 2014: 5 · 2013: 8 · 2012: 8
- FWCI
- 9.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- multimodal recognition, multimodal integration, robustness, semantic fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
