Multimodal social media video classification with deep neural networks
The DOI matches its Crossref record; 13 of the 15 references it cites were found in a registry or an index.
Abstract
Classifying videos according to their content is a common task across various contexts, as it allows effective content tagging, indexing and searching. In this work, we propose a general framework for video classification that is built on top of several neural network architectures. Since we rely on a multimodal approach, we extract both visual and textual features from videos and combine them in a final classification algorithm. When trained on a dataset of 30 000 social media videos and evaluated on 6 000 videos, our multimodal deep learning algorithm outperforms shallow single-modality classification methods by a large margin of up to 95%, achieving overall accuracy of 88%.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“a multimodal social media video classification pipeline leveraging both visual and textual features increased the classification accuracy to 88.0%, well above single modality neural networks such as Google’s InceptionV3 which reached an accuracy of 76.4% on the same task22.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [22]
In the citing paper
“Multimodal deep learning models that can ingest pixel data along with other data types (fusion) have been successful in applications outside of medicine, such as autonomous driving and video classification.”
From Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines (2020) · cites this paper as [22]
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299101 · Cited by 2,062
YouTube-8M: A Large-Scale Video Classification Benchmark
Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G. D., Varadarajan, B., & Vijayanarasimhan, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08675 · Cited by 848
Bags of Local Convolutional Features for Scalable Instance Search
Mohedano, E., McGuinness, K., O'Connor, N. E., Salvador, A., Marqués, F., & Giró-i-Nieto, X. (2016) · International Conference on Multimedia Retrieval · doi:10.1145/2911996.2912061 · Cited by 171
Multilayer and Multimodal Fusion of Deep Neural Networks for Video Classification
Yang, X., Molchanov, P., & Kautz, J. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2964297 · Cited by 112
Multimodal video classification with stacked contractive autoencoders
Liu, Y., Xiao-qing, F., & Zhou, Z. (2015) · Signal Processing · doi:10.1016/j.sigpro.2015.01.001 · Cited by 101
Multimodal Classification for Analysing Social MediaPreprint
Duong, C. T., Lebret, R., & Aberer, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02099 · Cited by 53
Video Classification With CNNs: Using the Codec as a Spatio-Temporal Activity Sensor
Chadha, A., Abbas, A., & Andreopoulos, Y. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2786999 · Cited by 44
Fusing Multi-Stream Deep Networks for Video ClassificationPreprint
Wu, Z., Jiang, Y., Wang, X., Ye, H., Xue, X., & Wang, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.06086 · Cited by 40
Research works printed but not linked: 1
- Wu, 2015, Fusing multi-stream deep networks for video classification, CoRR, abs/1509.06086 · arXiv
Cited by
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
Shetty, S., Ananthanarayana, V. S., & Mahale, A. R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14940-x · Cited by 37
Tang, P., Yan, X., Yang, N., Hu, X., Menze, B. H., Krammer, S., & Lasser, T. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110604 · Cited by 36
A Short Video Classification Framework Based on Cross-Modal Fusion
Pang, N., Guo, S., Yan, M. G., & Chan, C. A. (2023) · Sensors · doi:10.3390/s23208425 · Cited by 24
Single-sample image-fusion upsampling of fluorescence lifetime images
Kapitány, V., Fatima, A., Žičkus, V., Whitelaw, J., McGhee, E. J., Insall, R. H., Machesky, L. M., & Faccio, D. (2024) · Science Advances · doi:10.1126/sciadv.adn0139 · Cited by 12
Metadata
- Authors
- Trzcinski, T.
- Affiliations
- Warsaw University of Technology (PL)
- Corresponding author
- Trzcinski, Tomasz
- ORCID
- Trzcinski, Tomasz
- Venue
- Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2018
- Publisher
- SPIE
- Published
- 2018-10-01
- Pages
- 210
- OpenAlex
- W2894751678
- Cited by
- 6
- References
- 15
- Citations by year
- 2026: 1 · 2024: 3 · 2023: 1 · 2020: 1
- FWCI
- 0.08
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments (WILGA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- video classification, social media videos, multimodal fusion, visual features, text features, content tagging, video indexing, video retrieval, deep neural network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal social media video classification with deep neural networks. https://app.cochat.ai/papers/10.1117/12.2501679
- The paper
- Trzcinski, T. (2018). Multimodal social media video classification with deep neural networks. Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2018. https://doi.org/10.1117/12.2501679
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.