The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Towards Understanding Action Recognition
Abstract
Although action recognition in videos is widely studied, current methods often fail on real-world datasets. Many recent approaches improve accuracy and robustness to cope with challenging video sequences, but it is often unclear what affects the results most. This paper attempts to provide insights based on a systematic performance evaluation using thoroughly-annotated data of human actions. We annotate human Joints for the HMDB dataset (J-HMDB). This annotation can be used to derive ground truth optical flow and segmentation. We evaluate current methods using this dataset and systematically replace the output of various algorithms with ground truth. This enables us to discover what is important - for example, should we work on improving flow algorithms, estimating human bounding boxes, or enabling pose estimation? In summary, we find that high-level pose features greatly outperform low/mid level features, in particular, pose over time is critical, but current pose estimation algorithms are not yet reliable enough to provide this information. We also find that the accuracy of a top-performing action recognition framework can be greatly increased by refining the underlying low/mid level features, this suggests it is important to improve optical flow and human detection algorithms. Our analysis and J-HMDB dataset should facilitate a deeper understanding of action recognition algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
GrabCut: Interactive foreground extraction using iterated graph cuts
Rother, C., Kolmogorov, V., & Blake, A. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015720 · Cited by 5,703
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965
Self-citationHuman3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653 · cited 1×
“While marker-based motion capture systems provide accurate 3D ground-truth pose data [12, 15, 19, 25], they are impractical for recording realistic video data.”
BackgroundLearning realistic human actions from movies
Laptev, I., Marszalek, M., Schmid, C., & Rozenfeld, B. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587756 · Cited by 3,444
Self-citationTwo-Frame Motion Estimation Based on Polynomial Expansion
Farnebäck, G. (2003) · Lecture Notes in Computer Science · doi:10.1007/3-540-45103-x_50 · Cited by 2,971
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669
Self-citationHuman Detection Using Oriented Histograms of Flow and Appearance
Dalal, N., Triggs, B., & Schmid, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744047_33 · Cited by 1,652
Self-citationMarszalek, M., Laptev, I., & Schmid, C. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206557 · Cited by 1,168
Self-citation
Cited by
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.471 · Cited by 2,688
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
Xie, S., Sun, C., Huang, J., Tu, Z., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_19 · Cited by 1,415
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions
Gu, C., Sun, C., Ross, D. A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan, S., Toderici, G., Ricco, S., Sukthankar, R., et al. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00633 · Cited by 986
Self-citationLi, D., Opazo, C. R., Yu, X., & Li, H. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093512 · Cited by 641
P-CNN: Pose-Based CNN Features for Action Recognition
Cheron, G., Laptev, I., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.368 · Cited by 633
Self-citationCrowdPose: Efficient Crowded Scenes Pose Estimation and a New Benchmark
Li, J., Wang, C., Zhu, H., Mao, Y., Fang, H.-S., & Lu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01112 · Cited by 613
Human Action Recognition From Various Data Modalities: A Review
Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., & Liu, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183112 · Cited by 601
End-to-End Learning of Action Detection from Frame Glimpses in Videos
Yeung, S., Russakovsky, O., Mori, G., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.293 · Cited by 585
“2 shows our model structure.”
A Review of Human Activity Recognition Methods
Vrigkas, M., Nikou, C., & Kakadiaris, I. A. (2015) · Frontiers in Robotics and AI · doi:10.3389/frobt.2015.00028 · Cited by 566
Cross Modal Distillation for Supervision Transfer
Gupta, S., Hoffman, J., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.309 · Cited by 530
“Note that the representations ψi ∗ Ms and φ L Md may not have the same dimensions.”
Background2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning
Luvizon, D. C., Picard, D., & Tabia, H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00539 · Cited by 555
Unite the People: Closing the Loop Between 3D and 2D Human Representations
Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M. J., & Gehler, P. V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.500 · Cited by 541
Self-citationSkeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates
Liu, J., Shahroudy, A., Xu, D., Kot, A. C., & Wang, G. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2771306 · Cited by 530
Gkioxari, G., & Malik, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298676 · Cited by 528
PoseTrack: A Benchmark for Human Pose Estimation and Tracking
Andriluka, M., Iqbal, U., Insafutdinov, E., Pishchulin, L., Milan, A., Gall, J., & Schiele, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00542 · Cited by 486
Self-citationLearning Correspondence From the Cycle-Consistency of Time
Wang, X., Jabri, A., & Efros, A. A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00267 · Cited by 445
Monocular human pose estimation: A survey of deep learning-based methods
Chen, Y., Tian, Y., & He, M. (2020) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.102897 · Cited by 441
MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355414 · Cited by 424
Drone-surveillance for search and rescue in natural disaster
Mishra, B., Garg, D., Narang, P., & Mishra, V. (2020) · Computer Communications · doi:10.1016/j.comcom.2020.03.012 · Cited by 410
Driver Activity Recognition for Intelligent Vehicles: A Deep Learning Approach
Xing, Y., Lv, C., Wang, H., Cao, D., Velenis, E., & Wang, F.-Y. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2908425 · Cited by 399
Tracking Emerges by Colorizing Videos
Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_24 · Cited by 379
Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos
Hou, R., Chen, C., & Shah, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.620 · Cited by 375
Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark
Liang, X., Gong, K., Shen, X., & Lin, L. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2820063 · Cited by 365
Multi-region Two-Stream R-CNN for Action Detection
Peng, X., & Schmid, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_45 · Cited by 354
Self-citationFineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding
Shao, D., Zhao, Y., Dai, B., & Lin, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00269 · Cited by 353
Action Recognition with Stacked Fisher Vectors
Peng, X., Zou, C., Qiao, Y., & Peng, Q. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_38 · Cited by 338
PoTion: Pose MoTion Representation for Action Recognition
Choutas, V., Weinzaepfel, P., Revaud, J., & Schmid, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00734 · Cited by 332
Self-citationAction Tubelet Detector for Spatio-Temporal Action Localization
Kalogeiton, V., Weinzaepfel, P., Ferrari, V., & Schmid, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.472 · Cited by 321
Self-citationA Review on Deep Learning Techniques for Video Prediction
Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., & Argyros, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045007 · Cited by 301
Online Real-Time Multiple Spatiotemporal Action Localisation and Prediction
Singh, G., Saha, S., Sapienza, M., Torr, P., & Cuzzolin, F. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.393 · Cited by 279
Learning to Track for Spatio-Temporal Action Localization
Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.362 · Cited by 278
Self-citationA Survey on Deep Learning Technique for Video Segmentation
Zhou, T., Porikli, F., Crandall, D. J., Van Gool, L., & Wang, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3225573 · Cited by 255
Human action recognition using attention based LSTM network with dilated CNN features
Muhammad, K., Mustaqeem, Ullah, A., Imran, A. S., Sajjad, M., Kiran, M. S., Sannino, G., & de Albuquerque, V. H. C. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.06.045 · Cited by 254
(2016) · MPG.PuRe (Max Planck Society) · Cited by 107
Transformers for Vision: A Survey on Innovative Methods for Computer Vision
Palanisamy, B., Hassija, V., Chatterjee, A., Mandal, A., Chakraborty, D., Pandey, A., Chalapathi, G. S. S., & Kumar, D. (2025) · IEEE Access · doi:10.1109/access.2025.3571735 · Cited by 83
Body Joint Guided 3-D Deep Convolutional Descriptors for Action Recognition
Cao, C., Zhang, Y., Zhang, C., & Lu, H. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2756840 · Cited by 78
“Compared with Pose, HLPF used head instead of torso as the center to compute relative positions; HLPF also converted angles from”
BackgroundMethodResultInfluentialData-driven spatio-temporal RGBD feature encoding for action recognition in operating rooms
Twinanda, A. P., Alkan, E. O., Gangi, A., de Mathelin, M., & Padoy, N. (2015) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-015-1186-1 · Cited by 45
Hwang, I., Cha, G., & Oh, S. (2017) · 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI) · doi:10.1109/mfi.2017.8170441 · Cited by 28
Progressive modality cooperation for multi-modality domain adaptation
Zhang, W., Xu, D., Zhang, J., & Ouyang, W. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3052083 · Cited by 25
VideoMAC: Video Masked Autoencoders Meet ConvNets
Pei, G., Chen, T., Jiang, X., Liu, H., Sun, Z., & Yao, Y. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02145 · Cited by 23
“Notably, VideoMAC exhibits superior performance on a range of down-stream tasks, e.g ., video object segmentation on DAVIS17 [46], body part propagation on VIP [73], and human pose tracking on JHMDB [26], compared to ViT-based methods [16, 20].”
MethodInfluential
Metadata
- Authors
- Jhuang, H., Gall, J., Zuffi, S., Schmid, C., & Black, M. J.
- Venue
- 2013 IEEE International Conference on Computer Vision
- Publisher
- IEEE
- Published
- 2013-12
- Pages
- 3192-3199
- OpenAlex
- W2034014085
- Cited by
- 879
- References
- 36
- FWCI
- 20.27
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.