CoChat

Vision-based human action recognition: An overview and real world challenges

Jegham, Imen, Ben Khalifa, Anouar, Alouani, Ihsen, Mahjoub, Mohamed Ali · 2020 · Forensic Science International: Digital Investigation
Cited by 181

The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.

Abstract

Within a large range of applications in computer vision, Human Action Recognition has become one of the most attractive research fields. Ambiguities in recognizing actions does not only come from the difficulty to define the motion of body parts, but also from many other challenges related to real world problems such as camera motion, dynamic background, and bad weather conditions. There has been little research work in the real world conditions of human action recognition systems, which encourages us to seriously search in this application domain. Although a plethora of robust approaches have been introduced in the literature, they are still insufficient to fully cover the challenges. To quantitatively and qualitatively compare the performance of these methods, public datasets that present various actions under several conditions and constraints are recorded. In this paper, we investigate an overview of the existing methods according to the kind of issue they address. Moreover, we present a comparison of the existing datasets introduced for the human action recognition field.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.fsidi.2019.200901

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 102

  1. Aggarwal, 2011, Human activity analysis: a review, ACM Comput. Surv., 43, 16, 10.1145/1922649.1922653 · DOI · Paper page
  2. Aggarwal, 2014, Human activity recognition from 3d data: a review, Pattern Recognit. Lett., 48, 70, 10.1016/j.patrec.2014.04.011 · DOI · Paper page
  3. Ang, 2012, Wavelet mach filter for omnidirectional human activity recognition, Intl. Journal of Innovative Computing, Information and Control, 8, 1
  4. Anitha, 2019, Posture based health monitoring and unusual behavior recognition system for elderly using dynamic bayesian network, Cluster Computing, 13583, 10.1007/s10586-018-2010-9 · doi:10.1007/s10586-018-2010-9
  5. Baek, 2008, A sequence-action recognition applying state machine for user interface, IEEE Trans. Consum. Electron., 54, 719, 10.1109/TCE.2008.4560153 · doi:10.1109/tce.2008.4560153
  6. F. Baradel, C. Wolf, J. Mille, Pose-conditioned Spatio-Temporal Attention for Human Action Recognition, arXiv preprint arXiv:1703.10106. · arXiv
  7. Barnachon, 2014, Ongoing human action recognition with motion capture, Pattern Recognit., 47, 238, 10.1016/j.patcog.2013.06.020 · DOI
  8. Bobick, 1996, An appearance-based representation of action, vol. 1, 307
  9. Bobick, 2001, The recognition of human movement using temporal templates, IEEE Trans. Pattern Anal. Mach. Intell., 23, 257, 10.1109/34.910878 · DOI · Paper page
  10. Chaaraoui, 2013, Silhouette-based human action recognition using sequences of key poses, Pattern Recognit. Lett., 34, 1799, 10.1016/j.patrec.2013.01.021 · DOI
  11. Chakraborty, 2017, Review of constraints on vision-based gesture recognition for human–computer interaction, IET Comput. Vis., 12, 3, 10.1049/iet-cvi.2017.0052 · DOI · Paper page
  12. Chang, 2016, Nonparametric feature matching based conditional random fields for gesture recognition from multi-modal video, IEEE Trans. Pattern Anal. Mach. Intell., 38, 1612, 10.1109/TPAMI.2016.2519021 · DOI
  13. Chen, 2006, Meng Joo Er, Shiqian Wu, Illumination compensation and normalization for robust face recognition using discrete cosine transform in logarithm domain, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 36, 458, 10.1109/TSMCB.2005.857353 · DOI · Paper page
  14. Debes, 2016, Monitoring activities of daily living in smart homes: understanding human behavior, IEEE Signal Process. Mag., 33, 81, 10.1109/MSP.2015.2503881 · DOI · Paper page
  15. Duckworth, 2016, Unsupervised activity recognition using latent semantic analysis on a mobile robot, vol. 285, 1062
  16. Duta, 2017, Efficient human action recognition using histograms of motion gradients and vlad with descriptor shape information, Multimed. Tools Appl., 76, 22445, 10.1007/s11042-017-4795-6 · DOI
  17. Eraqi, 2019, Driver distraction identification with an ensemble of convolutional neural networks, Journal of Advanced Transportation, 2019, 1, 10.1155/2019/4125865 · DOI · Paper page
  18. Forsyth, 2006, Computational studies of human motion: part 1, tracking and motion synthesis, Found. Trends® Comput. Graph. Vis., 1, 77
  19. Gorelick, 2007, Actions as space-time shapes, IEEE Trans. Pattern Anal. Mach. Intell., 29, 2247, 10.1109/TPAMI.2007.70711 · DOI · Paper page
  20. Gorelick, 2007, Actions as space-time shapes, IEEE Trans. Pattern Anal. Mach. Intell., 29, 2247, 10.1109/TPAMI.2007.70711 · DOI · Paper page
  21. Guha, 2012, Learning sparse representations for human action recognition, IEEE Trans. Pattern Anal. Mach. Intell., 34, 1576, 10.1109/TPAMI.2011.253 · DOI · Paper page
  22. Guo, 2014, A survey on still image based human action recognition, Pattern Recognit., 47, 3343, 10.1016/j.patcog.2014.04.018 · DOI · Paper page
  23. Guo, 2013, Action recognition from video using feature covariance matrices, IEEE Trans. Image Process., 22, 2479, 10.1109/TIP.2013.2252622 · DOI
  24. Hadfield, 2017, Hollywood 3d: what are the best 3d features for action recognition?, Int. J. Comput. Vis., 121, 95, 10.1007/s11263-016-0917-2 · DOI
  25. He, 2017, Multi-view action recognition method based on regularized extreme learning machine, vol. 1, 854
  26. Hong, 2018, Not all frames are equal: aggregating salient features for dynamic texture classification, Multidimens. Syst. Signal Process., 29, 279, 10.1007/s11045-016-0463-7 · DOI
  27. Hu, 2017, Jointly learning heterogeneous features for rgb-d activity recognition, IEEE Trans. Pattern Anal. Mach. Intell., 39, 2186, 10.1109/TPAMI.2016.2640292 · DOI · Paper page
  28. Ikizler-Cinbis, 2012, Web-based classifiers for human action recognition, IEEE Trans. Multimed., 14, 1031, 10.1109/TMM.2012.2187180 · DOI
  29. Ishan Misra, 2016, Unsupervised learning using sequential verification for action recognition, Computer Science, Published in ArXiv
  30. Ji, 2010, Advances in view-invariant human motion analysis: a review, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 40, 13, 10.1109/TSMCC.2009.2027608 · DOI · Paper page
  31. Ji, 2013, Hidden markov model-based human action recognition using mixed features, J. Comput. Inf. Syst., 9, 3659
  32. Ji, 2019, A survey of human action analysis in hri applications, IEEE Trans. Circuits Syst. Video Technol.
  33. Jiang, 2006, Successive convex matching for action detection, vol. 2, 1646
  34. Jiang, 2012, Recognizing human actions by learning and matching shape-motion prototype trees, IEEE Trans. Pattern Anal. Mach. Intell., 34, 533, 10.1109/TPAMI.2011.147 · DOI
  35. Jun, 2013, Local transform features and hybridization for accurate face and human detection, IEEE Trans. Pattern Anal. Mach. Intell., 35, 1423, 10.1109/TPAMI.2012.219 · DOI
  36. Junejo, 2011, View-independent action recognition from temporal self-similarities, IEEE Trans. Pattern Anal. Mach. Intell., 33, 172, 10.1109/TPAMI.2010.68 · DOI · Paper page
  37. Kliper-Gross, 2012, The action similarity labeling challenge, IEEE Trans. Pattern Anal. Mach. Intell., 34, 615, 10.1109/TPAMI.2011.209 · DOI · Paper page
  38. Koppula, 2013, Learning human activities and object affordances from rgb-d videos, Int. J. Robot. Res., 32, 951, 10.1177/0278364913478446 · DOI · Paper page
  39. Kuehne, 2013, Hmdb51: a large video database for human motion recognition, vol. 12, 571
  40. Laptev, 2005, On space-time interest points, Int. J. Comput. Vis., 64, 107, 10.1007/s11263-005-1838-7 · DOI · Paper page
  41. Lavee, 2013, Propagating certainty in petri nets for activity recognition, IEEE Trans. Circuits Syst. Video Technol., 23, 326, 10.1109/TCSVT.2012.2203742 · doi:10.1109/tcsvt.2012.2203742
  42. Lei, 2018, A hierarchical representation for human action recognition in realistic scenes, Multimed. Tools Appl., 77, 11403, 10.1007/s11042-018-5626-0 · DOI
  43. Liu, 2010, Human action recognition using boosted eigenactions, Image Vis Comput., 28, 825, 10.1016/j.imavis.2009.07.009 · DOI
  44. Liu, 2015, Single/multi-view human action recognition via regularized multi-task learning, Neurocomputing, 151, 544, 10.1016/j.neucom.2014.04.090 · DOI
  45. Liu, 2015, Coupled hidden conditional random fields for rgb-d human action recognition, Signal Process., 112, 74, 10.1016/j.sigpro.2014.08.038 · DOI
  46. Liu, 2015, Multiple/single-view human action recognition via part-induced multitask structural learning, IEEE Transactions on Cybernetics, 45, 1194, 10.1109/TCYB.2014.2347057 · DOI
  47. Liu, 2016, Convolutional neural random fields for action recognition, Pattern Recognit., 59, 213, 10.1016/j.patcog.2016.03.019 · DOI
  48. Liu, 2017, Benchmarking a multimodal and multiview and interactive dataset for human action recognition, IEEE Transactions on Cybernetics, 47, 1781, 10.1109/TCYB.2016.2582918 · DOI · Paper page
  49. Lu, 2012, A human action recognition method based on tchebichef moment invariants and temporal templates, vol. 2, 76
  50. Lucas, 1981, An Iterative Image Registration Technique with an Application to Stereo Vision, vol. 2, 674
  51. Mansur, 2013, Inverse dynamics for action recognition, IEEE Transactions on Cybernetics, 43, 1226, 10.1109/TSMCB.2012.2226879 · DOI
  52. Minhas, 2010, Human action recognition using extreme learning machine based on visual vocabularies, Neurocomputing, 73, 1906, 10.1016/j.neucom.2010.01.020 · DOI
  53. Moeslund, 2006, A survey of advances in vision-based human motion capture and analysis, Comput. Vis. Image Understand., 104, 90, 10.1016/j.cviu.2006.08.002 · DOI · Paper page
  54. Nasiri, 2017, Human action recognition by fuzzy hidden markov model, Int. J. Comput. Vis. Robot., 7, 538, 10.1504/IJCVR.2017.086279 · doi:10.1504/ijcvr.2017.086279
  55. Nazir, 2018, Evaluating a bag-of-visual features approach using spatio-temporal features for action recognition, Comput. Electr. Eng., 72, 660, 10.1016/j.compeleceng.2018.01.037 · DOI
  56. Nez, 2018, Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition, Pattern Recognit., 76, 80, 10.1016/j.patcog.2017.10.033 · DOI · Paper page
  57. Niebles, 2008, Unsupervised learning of human action categories using spatial-temporal words, Int. J. Comput. Vis., 79, 299, 10.1007/s11263-007-0122-4 · DOI · Paper page
  58. Oikonomopoulos, 2006, Spatiotemporal salient points for visual recognition of human actions, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 36, 710, 10.1109/TSMCB.2005.861864 · DOI · Paper page
  59. Park, 2004, A hierarchical bayesian network for event recognition of human actions and interactions, Multimed. Syst., 10, 164, 10.1007/s00530-004-0148-1 · DOI · Paper page
  60. Peng, 2013, Exploring motion boundary based sampling and spatial-temporal context descriptors for action recognition, BMVC, 20, 93
  61. Quattoni, 2007, Hidden conditional random fields, IEEE Trans. Pattern Anal. Mach. Intell., 29, 1848, 10.1109/TPAMI.2007.1124 · DOI · Paper page
  62. Rahman, 2012, Recognising human actions by analysing negative spaces, IET Comput. Vis., 6, 197, 10.1049/iet-cvi.2011.0185 · DOI
  63. H. Rahmani, A. Mahmood, D. Huynh, A. Mian, Action Classification with Locality-Constrained Linear Coding, arXiv preprint arXiv:1408.3810. · arXiv
  64. Rahmani, 2016, Discriminative human action classification using locality-constrained linear coding, Pattern Recognit. Lett., 72, 62, 10.1016/j.patrec.2015.07.015 · DOI
  65. Rahmani, 2016, Histogram of oriented principal components for cross-view action recognition, IEEE Trans. Pattern Anal. Mach. Intell., 38, 2430, 10.1109/TPAMI.2016.2533389 · DOI · Paper page
  66. Rahmani, 2018, Learning a deep model for human action recognition from novel viewpoints, IEEE Trans. Pattern Anal. Mach. Intell., 40, 667, 10.1109/TPAMI.2017.2691768 · DOI · Paper page
  67. Ramanathan, 2014, Human action recognition with video data: research and evaluation challenges, IEEE Transactions on Human-Machine Systems, 44, 650, 10.1109/THMS.2014.2325871 · DOI · Paper page
  68. Reddy, 2013, Recognizing 50 human action categories of web videos, Mach. Vis. Appl., 24, 971, 10.1007/s00138-012-0450-4 · DOI · Paper page
  69. Schuldt, 2004, Recognizing human actions: a local svm approach, vol. 3, 32
  70. Shahroudy, 2016, Multimodal multipart learning for action recognition in depth videos, IEEE Trans. Pattern Anal. Mach. Intell., 38, 2123, 10.1109/TPAMI.2015.2505295 · DOI · Paper page
  71. Shao, 2012, Human action segmentation and recognition via motion and shape analysis, Pattern Recognit. Lett., 33, 438, 10.1016/j.patrec.2011.05.015 · DOI
  72. Shechtman, 2007, Space-time behavior-based correlation-or-how to tell if two underlying motion fields are similar without computing them?, IEEE Trans. Pattern Anal. Mach. Intell., 29, 2045, 10.1109/TPAMI.2007.1119 · DOI
  73. Sigal, 2010, Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion, Int. J. Comput. Vis., 87, 4, 10.1007/s11263-009-0273-6 · DOI · Paper page
  74. Singh, 2008, Human activity recognition based on silhouette directionality, IEEE Trans. Circuits Syst. Video Technol., 18, 1280, 10.1109/TCSVT.2008.928888 · DOI
  75. Sipiran, 2011, Harris 3d: a robust extension of the harris operator for interest point detection on 3d meshes, Vis. Comput., 27, 963, 10.1007/s00371-011-0610-y · DOI
  76. Song, 2014, Body surface context: a new robust feature for action recognition from depth videos, IEEE Trans. Circuits Syst. Video Technol., 24, 952, 10.1109/TCSVT.2014.2302558 · DOI
  77. Subramanian, 2012, Human action recognition using meta-cognitive neuro-fuzzy inference system, Int. J. Neural Syst., 22, 1250028, 10.1142/S0129065712500281 · DOI
  78. Sunny, 2015, Applications and challenges of human activity recognition using sensors in a smart environment, IJIRST Int. J. Innov. Res. Sci. Technol, 2, 50
  79. Suykens, 1999, Least squares support vector machine classifiers, Neural Process. Lett., 9, 293, 10.1023/A:1018628609742 · DOI · Paper page
  80. Turaga, 2008, Machine recognition of human activities: a survey, IEEE Trans. Circuits Syst. Video Technol., 18, 1473, 10.1109/TCSVT.2008.2005594 · DOI · Paper page
  81. Varol, 2018, Long-term temporal convolutions for action recognition, IEEE Trans. Pattern Anal. Mach. Intell., 40, 1510, 10.1109/TPAMI.2017.2712608 · DOI · Paper page
  82. Wang, 2013, Semantic context based refinement for news video annotation, Multimed. Tools Appl., 67, 607, 10.1007/s11042-012-1060-x · DOI
  83. Wang, 2018, Rgb-d-based human motion recognition with deep learning: a survey, Comput. Vis. Image Understand., 171, 118, 10.1016/j.cviu.2018.04.007 · DOI · Paper page
  84. Weinland, 2006, Free viewpoint action recognition using motion history volumes, Comput. Vis. Image Understand., 104, 249, 10.1016/j.cviu.2006.07.013 · DOI · Paper page
  85. Weinland, 2011, A survey of vision-based methods for action representation, segmentation and recognition, Comput. Vis. Image Understand., 115, 224, 10.1016/j.cviu.2010.10.002 · DOI · Paper page
  86. Wu, 2013, Silhouette analysis-based action recognition via exploiting human poses, IEEE Trans. Circuits Syst. Video Technol., 23, 236, 10.1109/TCSVT.2012.2203731 · DOI
  87. Wu, 2013, Action recognition using multilevel features and latent structural svm, IEEE Trans. Circuits Syst. Video Technol., 23, 1422, 10.1109/TCSVT.2013.2244794 · DOI
  88. Xiao, 2018, Action recognition based on hierarchical dynamic bayesian network, Multimed. Tools Appl., 77, 6955, 10.1007/s11042-017-4614-0 · DOI
  89. Xie, 2005, Face recognition under varying illumination based on a 2d face shape model, Pattern Recognit., 38, 221, 10.1016/S0031-3203(04)00275-4 · doi:10.1016/s0031-3203(04)00275-4
  90. Yang, 2013, Discovering motion primitives for unsupervised grouping and one-shot learning of human actions, gestures, and expressions, IEEE Trans. Pattern Anal. Mach. Intell., 35, 1635, 10.1109/TPAMI.2012.253 · DOI
  91. Yang, 2013, Rgb-depth feature for 3d human activity recognition, China Communications, 10, 93, 10.1109/CC.2013.6571292 · DOI
  92. Yeung, 2018, Every moment counts: dense detailed labeling of actions in complex videos, Int. J. Comput. Vis., 126, 375, 10.1007/s11263-017-1013-y · DOI · Paper page
  93. Yilmaz, 2005, Actions sketch: a novel action representation, vol. 1, 984
  94. Yilmaz, 2005, Recognizing human actions in videos acquired by uncalibrated moving cameras, vol. 1, 150
  95. Zaidenberg, 2014, Towards unsupervised sudden group movement discovery for video surveillance, vol. 2, 388
  96. Zelnik-Manor, 2001, Event-based analysis of video, vol. 2
  97. Zhang, 2017, A review on human activity recognition using vision-based method, Journal of healthcare engineering, 1
  98. Zhang, 2013, Auto learning temporal atomic actions for activity classification, Pattern Recognit., 46, 1789, 10.1016/j.patcog.2012.10.016 · doi:10.1016/j.patcog.2012.10.016
  99. Zhang, 2013, Modeling temporal interactions with interval temporal bayesian networks for complex activity recognition, IEEE Trans. Pattern Anal. Mach. Intell., 35, 2468, 10.1109/TPAMI.2013.33 · DOI
  100. Zhang, 2016, Rgb-d-based action recognition datasets: a survey, Pattern Recognit., 60, 86, 10.1016/j.patcog.2016.05.019 · DOI · Paper page
  101. Zhang, 2017, Semi-supervised image-to-video adaptation for video action recognition, IEEE Transactions on Cybernetics, 47, 960, 10.1109/TCYB.2016.2535122 · DOI
  102. H. Zhao, Z. Yan, H. Wang, L. Torresani, A. Torralba, Slac: A Sparsely Labeled Dataset for Action Classification and Localization, arXiv preprint arXiv:1712.09374. · arXiv

Cited by

See all 181

Metadata

Authors
Jegham, I., Ben Khalifa, A., Alouani, I., & Mahjoub, M. A.
Venue
Forensic Science International: Digital Investigation
Publisher
Elsevier BV
Published
2020-03
Volume
32
Pages
200901
ISSN
2666-2817 (ISSN-L)
OpenAlex
W3003646123
Cited by
181
References
19
FWCI
11.00
Type
journal article
Language
en
Open access
closed
Licence
  • www.elsevier.com/tdm/userlicense/1.0/ (tdm)
  • www.elsevier.com/legal/tdmrep-license (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.