The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index.
Optimal multimodal fusion for multimedia data analysis
Abstract
Considerable research has been devoted to utilizing multimodal features for better understanding multimedia data. However, two core research issues have not yet been adequately addressed. First, given a set of features extracted from multiple media sources (e.g., extracted from the visual, audio, and caption track of videos), how do we determine the best modalities? Second, once a set of modal-ities has been identified, how do we best fuse them to map to se-mantics? In this paper, we propose a two-step approach. The first step finds statistically independent modalities from raw features. In the second step, we use super-kernel fusion to determine the optimal combination of individual modalities. We carefully ana-lyze the tradeoffs between three design factors that affect fusion performance: modality independence, curse of dimensionality, and fusion-model complexity. Through analytical and empirical studies, we demonstrate that our two-step approach, which achieves a care-ful balance of the three design factors, can improve class-prediction accuracy over traditional techniques.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Attention-based fusion [49,50,51] aligns key features and dynamically assigns weights between modalities for deeper semantic capture.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [49]
Citations
Cites
An Information-Maximization Approach to Blind Separation and Blind Deconvolution
Bell, A. J., & Sejnowski, T. J. (1995) · Neural Computation · doi:10.1162/neco.1995.7.6.1129 · Cited by 8,986
Goodall, C. R., & Jolliffe, I. T. (1988) · Technometrics · doi:10.2307/1270093 · Cited by 8,797
Jolliffe, I. T. (1986) · Springer series in statistics · doi:10.1007/978-1-4757-1904-8 · Cited by 8,481
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 4,834
A Fast Fixed-Point Algorithm for Independent Component Analysis
Hyvärinen, A., & Erkki Oja, A. (1997) · Neural Computation · doi:10.1162/neco.1997.9.7.1483 · Cited by 3,373
Query by image and video content: the QBIC system
Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petković, D., et al. (1995) · Computer · doi:10.1109/2.410146 · Cited by 3,097
Adaptive Control Processes: A Guided Tour
Bellman, R. E. (1961) · doi:10.1515/9781400874668 · Cited by 2,802
Solving Multiclass Learning Problems via Error-Correcting Output Codes
Dietterich, T. G., & Bakiri, G. (1995) · Journal of Artificial Intelligence Research · doi:10.1613/jair.105 · Cited by 2,739
When Is ''Nearest Neighbor'' Meaningful?
Beyer, K., Goldstein, J., Ramakrishnan, R., & Shaft, U. (1999) · Lecture Notes in Computer Science · doi:10.1007/3-540-49257-7_15 · Cited by 2,403
A New Learning Algorithm for Blind Signal Separation
Амари, Ш.-И., Cichocki, A. S., & Yang, H. H. (1995) · Cited by 1,904
Research works printed but not linked: 10
- A. Amir , H. W, G. Iyengar , C.- Y. Lin , M. Naphade , A. Natsev , C. Neti , H. J. Nock , J. R. Smith , B. L. Tseng , Y. Wu , and D. Zhang . IBM research TRECVID-2003 system . NIST Text Retrieval Conf. (TREC) , 2003 .]] A. Amir, H. W, G. Iyengar, C.-Y.Lin, M. Naphade, A. Natsev, C. Neti, H. J. Nock, J. R. Smith, B. L. Tseng, Y. Wu, and D. Zhang. IBM research TRECVID-2003 system. NIST Text Retrieval Conf. (TREC), 2003.]]
- 10.5555/645503.656271 · DOI
- 10.5555/521772.791882 · DOI
- 10.5555/1622826.1622834 · DOI
- 10.5555/645496.658058 · DOI
- D. L. Donoho . High-dimensional data analysis: The curses and blessings of dimensionality . American Math. Society Lecture---Match Challenges of the 21st Century , 2000 .]] D. L. Donoho. High-dimensional data analysis: The curses and blessings of dimensionality. American Math. Society Lecture---Match Challenges of the 21st Century, 2000.]]
- Y. Rui , T. S. Huang , and S. F. Chang . Image retrieval: Past, present, and future . International Symposium on Multimedia Information Processing , 1997 .]] Y. Rui, T. S. Huang, and S. F. Chang. Image retrieval: Past, present, and future. International Symposium on Multimedia Information Processing, 1997.]]
- J. R. Smith and S. F. Chang . Automatic image retrieval using color and texture . IEEE Trans Pattern Anal Mach Intell , 1996 .]] J. R. Smith and S. F. Chang. Automatic image retrieval using color and texture. IEEE Trans Pattern Anal Mach Intell, 1996.]]
- 10.5555/1622859.1622868 · DOI
- 10.5555/1760167.1760197 · DOI
Cited by
Sorted by citations
Items 201–228 of 228
Giuliano Armano, B., Giuliani, A., Messina, A., Montagnuolo, M., & Vargiu, E. (2012) · Studies in computational intelligence · doi:10.1007/978-3-642-31546-6_3
Enhancing enterprise knowledge processes via cross-media extraction
Iria, J., Uren, V. S., Lavelli, A., Blohm, S., Dadzie, A., Franz, T., Kompatsiaris, I. Y., Magalhães, J., Nikolopoulos, S., Preisach, C., et al. (2007) · International Conference on Knowledge Capture · doi:10.1145/1298406.1298441
Per-Exemplar Fusion Learning for Video Retrieval and Recounting
Kim, I., Oh, S., Perera, A. G. A., & Lee, C. (2012) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2012.150
A conversation with Dr. Edward Y. Chang
Chang, E. Y. (2012) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2207243.2207256
Self-citationAHP: A New Strategy for the Semantic Concept Detection in Video
Ding, D., Zhang, B., & Wu, J. (2007) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2007.4285065
Extracting Multimedia Semantics Based on Independent Modality Discovering and Fusion
Xiao, R.-G., Zhuang, Y., & Wu, F. Q. (2006) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc.2006.258993
Access Control Based on RBAC in Distributed Cooperation Environment
Yin, S., & Wang, W. (2007) · International Symposium on Data, Privacy, and E-Commerce · doi:10.1109/isdpe.2007.14
Multimodal Discovering and Fusion for Semantics Multimedia Analysis
Hong, H. (2007) · International Conference on Advanced Language Processing and Web Information Technology · doi:10.1109/alpit.2007.111
A Multi-Pronged Approach to Improving Semantic Extraction of News Video
Hauptmann, A. G., Chen, M.-Y., Christel, M. G., Lin, W., & Yang, J. (2009) · Journal of Signal Processing Systems · doi:10.1007/s11265-009-0382-z
An Interactive Video Annotation Frameowrk with Multiple Modalities
Wang, M., Hua, X., Song, Y., Dai, L.-R., & Wang, R.-H. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366068
Mundle, A. R. (2012) · VTechWorks (Virginia Tech)
Hauptmann, A. G., Christel, M. G., & Yan, R. (2008)
Xhafa, F., Bader, D. A., Luca Benini, C., Bodin, F., Caballé, S., Jacques Demongeot, A., Doncescu, A., Drach, N., Curie, M., Gschwind, M. K., et al. (2008)
Web Image Semi-supervised Learning Method Based on Heterogeneous Information Fusion
Du, Y., Qian, L. I., Ya-Dong, Z. H. O. U., & Chen-He, W. U. (2012) · Acta Automatica Sinica · doi:10.3724/sp.j.1004.2012.01923
Banisakher, M. (2014) · Journal of International Crisis and Risk Communication Research
RETRACTED ARTICLE: Deeply fusing multimodal features in hypergraph
Yao, W., Jiang, Y., Jun, C., Xie, L., & Zhang, C. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-5305-6
RetractedMMDF2018 Workshop ReportPreprint
Chou, C.-A., Jin, X., Mueller, A. V., & Ostadabbas, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.10721
Kratkiewicz, K., Bear, J. W., & Miller, B. A. (2019)
Kenges, N., Ever, E., & Yazıcı, A. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-75078-7_25
Studying, developing, and experimenting contextual advertising systems
Giuliani, A. (2012) · UNICA IRIS Institutional Research Information System (University of Cagliari)
Indexation sémantique des images et des vidéos par apprentissage actif
Safadi, B. (2012) · theses.fr (ABES) · doi:10.70675/bbbde03ez6023z42cbzb31bz8428a0be1246
Research on the Method of Cross-modal Affective Association in Audiovisual
Jiang, J., & Zhang, H. (2021) · International Conference on Culture-oriented Science & Technology (ICCST) · doi:10.1109/iccst53801.2021.00019
Moulin, C. (2011) · theses.fr (ABES) · doi:10.70675/348762bcz4d33z4963zb271ze93744cef68c · Cited by 1
Apprentissage neuronal profond pour l'analyse de contenus multimodaux et temporels
Vielzeuf, V. (2019) · theses.fr (ABES) · doi:10.70675/cd4ec871zb372z492fz94bfz25086c5fda4a
Yue, Q., Wang, M., Xu, X., Long, C., Wang, Y., & Wang, J. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3786643
Semantic multimedia analysis using knowledge and context
Nikolopoulos, S. (2012)
Metadata
- Authors
- Wu, Y., Chang, E. Y., Chang, K. C.-C., & Smith, J. R.
- Affiliations
- University of California, Santa Barbara (US)
- University of Illinois Urbana-Champaign (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 12th annual ACM international conference on Multimedia
- Publisher
- ACM
- Published
- 2004-10-10
- Pages
- 572-579
- OpenAlex
- W2056952330
- Cited by
- 229
- References
- 35
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 2 · 2023: 2 · 2022: 3 · 2021: 5 · 2020: 5 · 2019: 8 · 2018: 5 · 2017: 6 · 2016: 7 · 2015: 9 · 2014: 12 · 2013: 15 · 2012: 24
- FWCI
- 14.19
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- National Science Foundation
- National Institute of Standards and Technology
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- multimodal fusion, feature fusion, modality selection, curse of dimensionality, video captioning, multimedia data analysis, class prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
