The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index.
Optimal multimodal fusion for multimedia data analysis
Abstract
Considerable research has been devoted to utilizing multimodal features for better understanding multimedia data. However, two core research issues have not yet been adequately addressed. First, given a set of features extracted from multiple media sources (e.g., extracted from the visual, audio, and caption track of videos), how do we determine the best modalities? Second, once a set of modal-ities has been identified, how do we best fuse them to map to se-mantics? In this paper, we propose a two-step approach. The first step finds statistically independent modalities from raw features. In the second step, we use super-kernel fusion to determine the optimal combination of individual modalities. We carefully ana-lyze the tradeoffs between three design factors that affect fusion performance: modality independence, curse of dimensionality, and fusion-model complexity. Through analytical and empirical studies, we demonstrate that our two-step approach, which achieves a care-ful balance of the three design factors, can improve class-prediction accuracy over traditional techniques.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Attention-based fusion [49,50,51] aligns key features and dynamically assigns weights between modalities for deeper semantic capture.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [49]
Citations
Cites
An Information-Maximization Approach to Blind Separation and Blind Deconvolution
Bell, A. J., & Sejnowski, T. J. (1995) · Neural Computation · doi:10.1162/neco.1995.7.6.1129 · Cited by 8,986
Goodall, C. R., & Jolliffe, I. T. (1988) · Technometrics · doi:10.2307/1270093 · Cited by 8,797
Jolliffe, I. T. (1986) · Springer series in statistics · doi:10.1007/978-1-4757-1904-8 · Cited by 8,481
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 4,834
A Fast Fixed-Point Algorithm for Independent Component Analysis
Hyvärinen, A., & Erkki Oja, A. (1997) · Neural Computation · doi:10.1162/neco.1997.9.7.1483 · Cited by 3,373
Query by image and video content: the QBIC system
Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petković, D., et al. (1995) · Computer · doi:10.1109/2.410146 · Cited by 3,097
Adaptive Control Processes: A Guided Tour
Bellman, R. E. (1961) · doi:10.1515/9781400874668 · Cited by 2,802
Solving Multiclass Learning Problems via Error-Correcting Output Codes
Dietterich, T. G., & Bakiri, G. (1995) · Journal of Artificial Intelligence Research · doi:10.1613/jair.105 · Cited by 2,739
When Is ''Nearest Neighbor'' Meaningful?
Beyer, K., Goldstein, J., Ramakrishnan, R., & Shaft, U. (1999) · Lecture Notes in Computer Science · doi:10.1007/3-540-49257-7_15 · Cited by 2,403
A New Learning Algorithm for Blind Signal Separation
Амари, Ш.-И., Cichocki, A. S., & Yang, H. H. (1995) · Cited by 1,904
Query by Image and Video Content: The QBIC System
Flickner, M., Sawhney, H., Niblack, W., Ashley, J. J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petković, D., et al. (2002) · Elsevier eBooks · doi:10.1016/b978-155860651-7/50108-x · Cited by 1,559
Optimal aggregation algorithms for middleware
Fagin, R., Lotem, A., & Naor, M. (2001) · doi:10.1145/375551.375567 · Cited by 1,090
Bellman, R. (1964) · IEEE Transactions on Circuit Theory · doi:10.1109/tct.1964.1082221 · Cited by 1,003
A min-max cut algorithm for graph partitioning and data clustering
Ding, C. H. Q., He, X., Hongyuan Zha, B., Gu, M., & Simon, H. D. (2002) · Proceedings · doi:10.1109/icdm.2001.989507 · Cited by 850
Content-based image retrieval with relevance feedback in MARS
Rui, Y., Huang, T. S., & Mehrotra, S. (2002) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.1997.638621 · Cited by 729
(2000) · Applied Physics Letters · doi:10.1162/153244303768966085 · Cited by 583
Issues in Stacked Generalization
Ting, K. M., & Witten, I. H. (1999) · Cited by 521
Combining multiple classifiers by averaging or by multiplying?
Tax, D. M. J., van Breukelen, M., Duin, R. P. W., & Kittler, J. (2000) · Pattern Recognition · doi:10.1016/s0031-3203(99)00138-7 · Cited by 377
Independent component representations for face recognition
Bartlett, M. S., Lades, M. H., & B, T. J. S. A. (1998) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.320144 · Cited by 316
Combining textual and visual cues for content-based image retrieval on the World Wide Web
Cascia, M. L., Sethi, S., & Sclaroff, S. (2002) · doi:10.1109/ivl.1998.694480 · Cited by 249
Audio Vision: Using Audio-Visual Synchrony to Locate Sounds
Hershey, J. R., & Movellan, J. R. (1999) · Cited by 227
Inferring a Semantic Representation of Text via Cross-Language Correlation Analysis
Vinokourov, A., Cristianini, N., & Shawe-Taylor, J. S. (2002) · Cited by 225
Kittler, J., Hater, M., & Duin, R. P. W. (1996) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1996.547205 · Cited by 200
Learning Joint Statistical Models for Audio-Visual Fusion and Segregation
Fisher, J. S., Darrell, T., Freeman, W. T., & Viola, P. A. (2000) · Cited by 176
Image retrieval: content versus context
Westerveld, T. (2000) · University of Twente Research Information · Cited by 93
SVM binary classifier ensembles for image classification
Goh, K.-S., Chang, E. Y., & Cheng, K.-T. T. (2001) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/502585.502652 · Cited by 86
Self-citationDiscovery of a perceptual distance function for measuring image similarity
Li, B., Chang, E. Y., & Wu, Y. (2003) · Multimedia Systems · doi:10.1007/s00530-002-0069-9 · Cited by 77
Self-citationAUDIO/VISUAL INDEPENDENT COMPONENTS
Smaragdis, P., Casey, M. A., & Square, N. (2003) · Cited by 66
Independent component analysis for understanding multimedia content
Kolenda, T., Hansen, L. K., Larsen, J. O., & Winther, O. (2003) · Machine learning for signal processing ... · doi:10.1109/nnsp.2002.1030096 · Cited by 57
The combination limit in multimedia retrieval
Yan, R., & Hauptmann, A. G. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957013.957086 · Cited by 56
On Independent Component Analysis for Multimedia Signals
Kolenda, T., Larsen, J. O., & Hansen, L. K. (2000) · Image processing series · doi:10.1201/9781420037562.ch7 · Cited by 34
Vinokourov, A., Hardoon, D. R., & Shawe-Taylor, J. S. (2003) · ePrints Soton (University of Southampton) · Cited by 32
Detection of Documentary Scene Changes by Audio-Visual Fusion
Velivelli, A., Ngo, C., & Huang, T. S. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45113-7_23 · Cited by 15
An ICA algorithm for analyzing multiple data sets
Lukic, A. S., Wernick, M. N., Hansen, L. K., & Strother, S. C. (2003) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2002.1040077 · Cited by 13
Research works printed but not linked: 10
- A. Amir , H. W, G. Iyengar , C.- Y. Lin , M. Naphade , A. Natsev , C. Neti , H. J. Nock , J. R. Smith , B. L. Tseng , Y. Wu , and D. Zhang . IBM research TRECVID-2003 system . NIST Text Retrieval Conf. (TREC) , 2003 .]] A. Amir, H. W, G. Iyengar, C.-Y.Lin, M. Naphade, A. Natsev, C. Neti, H. J. Nock, J. R. Smith, B. L. Tseng, Y. Wu, and D. Zhang. IBM research TRECVID-2003 system. NIST Text Retrieval Conf. (TREC), 2003.]]
- 10.5555/645503.656271 · DOI
- 10.5555/521772.791882 · DOI
- 10.5555/1622826.1622834 · DOI
- 10.5555/645496.658058 · DOI
- D. L. Donoho . High-dimensional data analysis: The curses and blessings of dimensionality . American Math. Society Lecture---Match Challenges of the 21st Century , 2000 .]] D. L. Donoho. High-dimensional data analysis: The curses and blessings of dimensionality. American Math. Society Lecture---Match Challenges of the 21st Century, 2000.]]
- Y. Rui , T. S. Huang , and S. F. Chang . Image retrieval: Past, present, and future . International Symposium on Multimedia Information Processing , 1997 .]] Y. Rui, T. S. Huang, and S. F. Chang. Image retrieval: Past, present, and future. International Symposium on Multimedia Information Processing, 1997.]]
- J. R. Smith and S. F. Chang . Automatic image retrieval using color and texture . IEEE Trans Pattern Anal Mach Intell , 1996 .]] J. R. Smith and S. F. Chang. Automatic image retrieval using color and texture. IEEE Trans Pattern Anal Mach Intell, 1996.]]
- 10.5555/1622859.1622868 · DOI
- 10.5555/1760167.1760197 · DOI
Cited by
Sorted by citations
Datta, R., Joshi, D., Li, J., & Wang, J. Z. (2008) · ACM Computing Surveys · doi:10.1145/1348246.1348248 · Cited by 3,080
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
Early versus late fusion in semantic video analysis
Snoek, C. G. M., Worring, M., & Smeulders, A. W. M. (2005) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1101149.1101236 · Cited by 899
Cross-domain video concept detection using adaptive svms
Yang, J. J., Yan, R., & Hauptmann, A. G. (2007) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1291233.1291276 · Cited by 673
Unified Video Annotation via Multigraph Learning
Wang, M., Hua, X., Hong, R., Tang, J., Qi, G.-J., & Song, Y. (2009) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2009.2017400 · Cited by 443
Metadata
- Authors
- Wu, Y., Chang, E. Y., Chang, K. C.-C., & Smith, J. R.
- Affiliations
- University of California, Santa Barbara (US)
- University of Illinois Urbana-Champaign (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 12th annual ACM international conference on Multimedia
- Publisher
- ACM
- Published
- 2004-10-10
- Pages
- 572-579
- OpenAlex
- W2056952330
- Cited by
- 229
- References
- 35
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 2 · 2023: 2 · 2022: 3 · 2021: 5 · 2020: 5 · 2019: 8 · 2018: 5 · 2017: 6 · 2016: 7 · 2015: 9 · 2014: 12 · 2013: 15 · 2012: 24
- FWCI
- 14.19
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- National Science Foundation
- National Institute of Standards and Technology
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- multimodal fusion, feature fusion, modality selection, curse of dimensionality, video captioning, multimedia data analysis, class prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
