The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
Open accessAbstract
In this paper, we study the challenging problem of categorizing videos according to high-level semantics such as the existence of a particular human action or a complex event. Although extensive efforts have been devoted in recent years, most existing works combined multiple video features using simple fusion strategies and neglected the utilization of inter-class semantic relationships. This paper proposes a novel unified framework that jointly exploits the feature relationships and the class relationships for improved categorization performance. Specifically, these two types of relationships are estimated and utilized by imposing regularizations in the learning process of a deep neural network (DNN). Through arming the DNN with better capability of harnessing both the feature and the class relationships, the proposed regularized DNN (rDNN) is more suitable for modeling video semantics. We show that rDNN produces better performance over several state-of-the-art approaches. Competitive results are reported on the well-known Hollywood2 and Columbia Consumer Video benchmarks. In addition, to stimulate future research on large scale video categorization, we collect and release a new benchmark dataset, called FCVID, which contains 91,223 Internet videos and 239 manually annotated categories.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965
Learning realistic human actions from movies
Laptev, I., Marszalek, M., Schmid, C., & Rozenfeld, B. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587756 · Cited by 3,444
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Recognizing human actions: A local SVM approach
Schüldt, C., Laptev, I., & Barbara Caputo, B. (2004) · Cited by 2,908
(2011) · Cited by 2,752
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.471 · Cited by 2,688
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.213 · Cited by 2,684
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299101 · Cited by 2,062
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861
Blank, M., Gorelick, L., Shechtman, E., Irani, M., & Basri, R. (2005) · Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 · doi:10.1109/iccv.2005.28 · Cited by 1,733
Multisensory integration: current issues from the perspective of the single neuron
Stein, B. E., & Stanford, T. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2331 · Cited by 1,598
Convex multi-task feature learning
Argyriou, A., Evgeniou, T., & Pontil, M. (2008) · Machine Learning · doi:10.1007/s10994-007-5040-8 · Cited by 1,528
Image Classification with the Fisher Vector: Theory and Practice
Sánchez, J., Perronnin, F., Mensink, T., & Verbeek, J. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0636-x · Cited by 1,498
Multiple kernel learning, conic duality, and the SMO algorithm
Bach, F. R., Lanckriet, G. R. G., & Jordan, M. I. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015424 · Cited by 1,453
Large-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,386
Three things everyone should know to improve object retrieval
Arandjelovic, R., & Zisserman, A. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248018 · Cited by 1,383
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
Classification using intersection kernel support vector machines is efficient
Maji, S., Berg, A. C., & Malik, J. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587630 · Cited by 940
Contextual Priming for Object Detection
Torralba, A. (2003) · International Journal of Computer Vision · doi:10.1023/a:1023052124951 · Cited by 776
Multiple kernels for object detection
Vedaldi, A., Gulshan, V., Varma, M., & Zisserman, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459183 · Cited by 757
A Biologically Inspired System for Action Recognition
Jhuang, H., Serre, T., Wolf, L., & Poggio, T. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408988 · Cited by 723
Modeling Temporal Structure of Decomposable Motion Segments for Activity Classification
Niebles, J. C., Chen, C.-W., & Fei-Fei, L. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15552-9_29 · Cited by 700
Zhang, D., & Shen, D. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.09.069 · Cited by 689
Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., & Belongie, S. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408986 · Cited by 594
Modeling video evolution for action recognition
Fernando, B., Gavves, E., Jose Oramas, M., Ghodrati, A., & Tuytelaars, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299176 · Cited by 569
Multi-Task Feature Learning Via Efficient l2,1-Norm Minimization
Liu, J., Ji, S., & Ye, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2631 · Cited by 555
Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification
Wu, Z., Wang, X., Jiang, Y.-G., Ye, H., & Xue, X. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2806222 · Cited by 479
Self-citationCorrelative multi-label video annotation
Qi, G.-J., Hua, X.-S., Rui, Y., Tang, J., Mei, T., & Zhang, H.-J. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291245 · Cited by 477
lp-Norm Multiple Kernel Learning
Kloft, M. (2011) · DepositOnce · doi:10.14279/depositonce-2978 · Cited by 412
Clustered Multi-Task Learning: A Convex Formulation
Jacob, L., Bach, F., & Vert, J. (2008) · arXiv (Cornell University) · doi:10.48550/arxiv.0809.2085 · Cited by 400
Better Exploiting Motion for Better Action Recognition
Jain, M., Jegou, H., & Bouthemy, P. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.330 · Cited by 385
Large-Scale Object Classification Using Label Relation Graphs
Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H., & Adam, H. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_4 · Cited by 347
A convex formulation for learning task relationships in multi-task learning
Zhang, Y., & Yeung, D. (2010) · Cited by 338
Jiang, Y.-G., Ye, G., Chang, S.-F., Ellis, D., & Loui, A. C. (2011) · Proceedings of the 1st ACM International Conference on Multimedia Retrieval · doi:10.1145/1991996.1992025 · Cited by 324
Self-citationA normalization model of multisensory integration
Ohshiro, T., Angelaki, D. E., & DeAngelis, G. C. (2011) · Nature Neuroscience · doi:10.1038/nn.2815 · Cited by 320
Learning with Whom to Share in Multi-task Feature Learning
Kang, Z., Grauman, K. L., & Sha, F. (2011) · Cited by 320
Integrating low-rank and group-sparse structures for robust multi-task learning
Chen, J., Zhou, J., & Ye, J. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020423 · Cited by 311
Beyond Gaussian Pyramid: Multi-skip Feature Stacking for action recognition
Lan, Z., Lin, M., Li, X., Hauptmann, A. G., & Raj, B. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298616 · Cited by 264
A multi-task learning formulation for predicting disease progression
Zhou, J., Yuan, L., Liu, J., & Ye, J. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020549 · Cited by 254
A probabilistic framework for semantic video indexing, filtering, and retrieval
Naphide, H. R., & Huang, T. S. (2001) · IEEE Transactions on Multimedia · doi:10.1109/6046.909601 · Cited by 240
COSTA: Co-Occurrence Statistics for Zero-Shot Classification
Mensink, T., Gavves, E., & Snoek, C. G. M. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.313 · Cited by 193
Robust late fusion with rank minimization
Ye, G., Liu, D., Jhuo, I.-H., & Chang, S.-F. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248032 · Cited by 189
Self-citationMultimodal feature fusion for robust event detection in web videos
Natarajan, P., Wu, S., Vitaladevuni, S., Zhuang, X., Tsakalidis, S., Park, U., Prasad, R., & Natarajan, P. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247814 · Cited by 178
Wu, Z., Jiang, Y., Wang, J., Pu, J., & Xue, X. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654931 · Cited by 156
Self-citationImproved Multimodal Deep Learning with Variation of Information
Sohn, K., Shang, W., & Lee, H. (2014) · neural information processing systems · Cited by 145
Ye, G., Li, Y., Xu, H., Liu, D., & Chang, S. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806221 · Cited by 128
Self-citationMulti-GPU Training of ConvNetsPreprint
Yadan, O., Adams, K., Taigman, Y., & Ranzato, M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.5853 · Cited by 98
Heterogeneous feature machines for visual recognition
Cao, L., Luo, J., Liang, F., & Huang, T. S. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459401 · Cited by 84
Improving Human Action Recognition Using Score Distribution and Ranking
Hoai, M., & Zisserman, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16814-2_1 · Cited by 81
Large-Scale Video Hashing via Structure Learning
Ye, G., Liu, D., Wang, J., & Chang, S. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.282 · Cited by 80
Self-citationMulti-Task Learning for Stock Selection
Ghosn, J., & Bengio, Y. (1996) · Cited by 76
Sample-Specific Late Fusion for Visual Category Recognition
Liu, D., Lai, K.-T., Ye, G., et al. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.109 · Cited by 68
Short-term audio-visual atoms for generic video concept classification
Jiang, W., Cotton, C. V., Chang, S., et al. (2009) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1631272.1631277 · Cited by 67
Self-citationFeature Weighting via Optimal Thresholding for Video Analysis
Xu, Z., Yang, Y., Tsang, I. W.-H., Sebe, N., & Hauptmann, A. G. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.427 · Cited by 50
Mariana: tencent deep learning platform and its applications
Zou, Y., Jin, X., Li, Y., Guo, Z., Wang, E., & Xiao, B. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2733004.2733082 · Cited by 49
Fast Semantic Diffusion for Large-Scale Context-Based Image and Video Annotation
Jiang, Y., Dai, Q., Wang, J., Ngo, C., Xue, X., & Chang, S. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2188038 · Cited by 49
Self-citationDiscovering joint audio–visual codewords for video event detection
Jhuo, I., Ye, G., Gao, S., Liu, D., Jiang, Y., Lee, D. T., & Chang, S. (2013) · Machine Vision and Applications · doi:10.1007/s00138-013-0567-0 · Cited by 46
Self-citationSimplex-Based 3D Spatio-temporal Feature Description for Action Recognition
Zhang, H., Zhou, W., Reardon, C., et al. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.265 · Cited by 42
Structured feature selection and task relationship inference for multi-task learning
Fei, H., & Huan, J. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-012-0543-4 · Cited by 37
Video Classification Using Semantic Concept Co-occurrences
Assari, S. M., Zamir, A., & Shah, M. A. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.324 · Cited by 34
A novel audio fingerprinting method robust to time scale modification and pitch shifting
Zhu, B., Li, W., Wang, Z., et al. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1874130 · Cited by 33
CMU-Informedia @ TRECVID 2013 Multimedia Event Detection
Jiang, L., Lan, Z., Yu, S.-I., Cai, Y., Gao, C., Xu, S., Shen, H., Wang, Y., Sze, W., Yan, Y., et al. (2013) · Cited by 27
Jiang, Y. (2012) · International Conference on Multimedia Retrieval · doi:10.1145/2324796.2324805 · Cited by 26
Self-citationJiang, W., & Loui, A. C. P. (2011) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2072298.2072316 · Cited by 25
Reduced Analytic Dependency Modeling: Robust Fusion for Visual Recognition
Ma, A. J., & Yuen, P. C. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0723-7 · Cited by 25
Cross-Domain Multicue Fusion for Concept-Based Video Indexing
Weng, M.-F., & Chuang, Y. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.273 · Cited by 23
Using Web Co-occurrence Statistics for Improving Image CategorizationPreprint
Samy Bengio, A., Dean, J., Erhan, D., Ie, E., Le, Q., Rabinovich, A., Shlens, J., & Singer, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.5697 · Cited by 22
The AXES submissions at TrecVid 2013
Aly, R., Arandjelović, R., Chatfield, K., Douze, M., Fernando, B., Harchaoui, Z., McGuinness, K., O'Connor, N. E., Oneaţă, D., Parkhi, O., et al. (2013) · Lirias · Cited by 20
Beta Process Multiple Kernel Learning
Ni, B., Li, T., & Moulin, P. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.128 · Cited by 8
Subband autocorrelation features for video soundtrack classification
Cotton, C. V., & Ellis, D. P. W. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639357 · Cited by 6
Cited by
Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
Qiu, Z., Yao, T., & Mei, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.590 · Cited by 1,874
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
YouTube-8M: A Large-Scale Video Classification Benchmark
Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G. D., Varadarajan, B., & Vijayanarasimhan, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08675 · Cited by 848
Human Action Recognition and Prediction: A Survey
Kong, Y., & Fu, Y. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01594-9 · Cited by 780
Metadata
- Authors
- Jiang, Y.-G., Wu, Z., Wang, J., Xue, X., & Chang, S.-F.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2018-02-01
- Volume
- 40
- Issue
- 2
- Pages
- 352-364
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W1777628566
- Cited by
- 381
- References
- 80
- FWCI
- 20.33
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.