The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Early versus late fusion in semantic video analysis
Open accessAbstract
Semantic analysis of multimodal video aims to index segments of interest at a conceptual level. In reaching this goal, it requires an analysis of several information streams. At some point in the analysis these streams need to be fused. In this paper, we consider two classes of fusion schemes, namely early fusion and late fusion. The former fuses modalities in feature space, the latter fuses modalities in semantic space. We show by experiment on 184 hours of broadcast video data and for 20 semantic concepts, that late fusion tends to give slightly better performance for most concepts. However, for those concepts where early fusion performs better the difference is more significant.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Platt, J. C. (2000) · Advances in Large-Margin Classifiers · doi:10.7551/mitpress/1113.003.0008 · Cited by 1,156
The LIMSI Broadcast News transcription system
Gauvain, J., Lamel, L. F., & Adda, G. (2002) · Speech Communication · doi:10.1016/s0167-6393(01)00061-9 · Cited by 433
Optimal multimodal fusion for multimedia data analysis
Wu, Y., Chang, E. Y., Chang, K. C.-C., & Smith, J. R. (2004) · Proceedings of the 12th annual ACM international conference on Multimedia · doi:10.1145/1027527.1027665 · Cited by 229
Content-based video parsing and indexing based on audio-visual interaction
Tsekeridou, S., & Pitas, I. (2001) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/76.915358 · Cited by 74
Discriminative model fusion for semantic concept detection and annotation in video
Iyengar, G., & Nock, H. J. (2003) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/957013.957065 · Cited by 69
A Probabilistic Multimedia Retrieval Model and Its Evaluation
Westerveld, T., de Vries, A. P., van Ballegooij, A., de Jong, F. M. G., & Hiemstra, D. (2003) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s111086570321101x · Cited by 63
Cited by
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
Evaluation campaigns and TRECVid
Smeaton, A. F., Over, P., & Kraaij, W. (2006) · Proceedings of the 8th ACM international workshop on Multimedia information retrieval · doi:10.1145/1178677.1178722 · Cited by 1,224
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853
Recognizing 50 human action categories of web videos
Reddy, K. K., & Shah, M. (2013) · Machine Vision and Applications · doi:10.1007/s00138-012-0450-4 · Cited by 655
The challenge problem for automated detection of 101 semantic concepts in multimedia
Snoek, C. G. M., Worring, M., van Gemert, J. C., Geusebroek, J.-M., & Smeulders, A. W. M. (2006) · Proceedings of the 14th ACM international conference on Multimedia · doi:10.1145/1180639.1180727 · Cited by 620
Self-citationA Survey on Visual Content-Based Video Indexing and Retrieval
Hu, W., Xie, N., Li, L., Zeng, X., & Maybank, S. (2011) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2011.2109710 · Cited by 585
A Review of Human Activity Recognition Methods
Vrigkas, M., Nikou, C., & Kakadiaris, I. A. (2015) · Frontiers in Robotics and AI · doi:10.3389/frobt.2015.00028 · Cited by 566
Click Prediction for Web Image Reranking Using Multimodal Sparse Coding
Yu, J., Rui, Y., & Tao, D. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2311377 · Cited by 550
COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios
Pereira, R. M., Bertolini, D., Teixeira, L. O., Silla, C. N., & Costa, Y. M. G. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105532 · Cited by 509
3D Object Detection for Autonomous Driving: A Survey
Qian, R., Lai, X., & Li, X. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108796 · Cited by 454
Unified Video Annotation via Multigraph Learning
Wang, M., Hua, X.-S., Hong, R., Tang, J., Qi, G.-J., & Song, Y. (2009) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2009.2017400 · Cited by 441
A discriminative CNN video representation for event detection
Xu, Z., Yang, Y., & Hauptmann, A. G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298789 · Cited by 435
Verma, G. K., & Tiwary, U. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.007 · Cited by 354
Multimodal Graph-Based Reranking for Web Image Search
Wang, M., Li, H., Tao, D., Lu, K., & Wu, X. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2207397 · Cited by 344
Kim, T., & Kim, H. Y. (2019) · PLOS ONE · doi:10.1371/journal.pone.0212320 · Cited by 342
Multiple feature hashing for real-time large scale near-duplicate video retrieval
Song, J., Yang, Y., Huang, Z., Shen, H. T., & Hong, R. (2011) · Proceedings of the 19th ACM international conference on Multimedia · doi:10.1145/2072298.2072354 · Cited by 334
“Besides, as indicated in [20], late fusion is computationally more expensive for training.”
BackgroundSnoek, C. G. M., & Worring, M. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601982353 · Cited by 317
Self-citationBeyond Distance Measurement: Constructing Neighborhood Similarity for Video Annotation
Wang, M., Hua, X.-S., Tang, J., & Hong, R. (2009) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2009.2012919 · Cited by 288
Multimodal Deep Learning for Activity and Context Recognition
Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161174 · Cited by 281
Color attributes for object detection
Shahbaz Khan, F., Anwer, R. M., van de Weijer, J., Bagdanov, A. D., Vanrell, M., & Lopez, A. M. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248068 · Cited by 264
High-Order Distance-Based Multiview Stochastic Learning in Image Classification
Yu, J., Rui, Y., Tang, Y. Y., & Tao, D. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2307862 · Cited by 260
G-MS2F: GoogLeNet based multi-stage feature fusion of deep CNN for scene recognition
Tang, P., Wang, H., & Kwong, S. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.11.023 · Cited by 259
Effective Multiple Feature Hashing for Large-Scale Near-Duplicate Video Retrieval
Song, J., Yang, Y., Huang, Z., Shen, H. T., & Luo, J. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2271746 · Cited by 257
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204
Gosztolya, G., Vincze, V., Tóth, L., Pákáski, M., Kálmán, J., & Hoffmann, I. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2018.07.007 · Cited by 196
MFAS: Multimodal Fusion Architecture Search
Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00713 · Cited by 193
Learn to combine modalities in multimodal deep learning
Liu, K., Li, Y., Xu, N., & Natarajan, P. (2018) · arXiv · doi:10.48550/arxiv.1805.11730 · Cited by 136
“This idea is ubiquitous in existing multimodal techniques, including early and late fusion [42, 15], hybrid fusion [1], model ensemble [7], and more recently—joint training methods based on deep neural networks [38, 45, 37].”
BackgroundMultimodal mental health analysis in social media
Yazdavar, A. H., Mahdavinejad, M. S., Bajaj, G., Romine, W., Sheth, A., Monadjemi, A. H., Thirunarayan, K., Meddar, J. M., Myers, A., Pathak, J., et al. (2020) · PLOS ONE · doi:10.1371/journal.pone.0226248 · Cited by 133
Artificial Intelligence and Data Fusion at the Edge
Munir, A., Blasch, E., Kwon, J., Kong, J., & Aved, A. (2021) · IEEE Aerospace and Electronic Systems Magazine · doi:10.1109/maes.2020.3043072 · Cited by 110
“…at fusion architectures that emphasize fusion-focused data analysis such as the data-feature-decision (DFD) [12] model, wireless sensor networks [14], and the more recent efforts from the AI community with the early and late fusion (ELF) methods [15] and generative adversarial networks (GANs).”
BackgroundAdeel, A., Gogate, M., & Hussain, A. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.08.008 · Cited by 96
“Researchers have proposed novel visual feature extraction methods [9][10][11][12][13], fusion approaches (early integration [14], late integration [15], hybrid integration [16]), multi-modal datasets [17][18], and fusion techniques [16][19][20].”
BackgroundAutomatic detection of articulation disorders in children with cleft lip and palate
Maier, A., Hönig, F., Bocklet, T., Nöth, E., Stelzle, F., Nkenke, E., & Schuster, M. (2009) · The Journal of the Acoustical Society of America · doi:10.1121/1.3216913 · Cited by 71
A Comparative Analysis of Early and Late Fusion for the Multimodal Two-Class Problem
Pereira, L. M., Salazar, A., & Vergara, L. (2023) · IEEE Access · doi:10.1109/access.2023.3296098 · Cited by 69
(2019) · doi:10.1109/smc.2019.8914655 · Cited by 52
Event-based media processing and analysis: A survey of the literature
Tzelepis, C., Ma, Z., Mezaris, V., Ionescu, B., Kompatsiaris, I., Boato, G., Sebe, N., & Yan, S. (2016) · Image and Vision Computing · doi:10.1016/j.imavis.2016.05.005 · Cited by 49
“This is more consistent with human’s understanding and reasoning about the task, where an event is characterized by the presence/absence of certain concepts rather than interest points [94, 181, 114].”
BackgroundOn the Generality of Codebook Approach for Sensor-Based Human Activity Recognition
Shirahama, K., & Grzegorzek, M. (2017) · Electronics · doi:10.3390/electronics6020044 · Cited by 29
Dimitri, G. M. (2022) · Computers · doi:10.3390/computers11110163 · Cited by 22
Predicting the Popularity of Micro-videos with Multimodal Variational Encoder-Decoder Framework
Zhu, Y., Xie, J., & Chen, Z. (2020) · arXiv · doi:10.48550/arxiv.2003.12724 · Cited by 5
Metadata
- Authors
- Snoek, C. G. M., Worring, M., & Smeulders, A. W. M.
- Venue
- Proceedings of the 13th annual ACM international conference on Multimedia
- Publisher
- ACM
- Published
- 2005-11-06
- Pages
- 399-402
- OpenAlex
- W1989085630
- Cited by
- 879
- References
- 7
- FWCI
- 15.89
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.