Event-based Action Recognition Using Motion Information and Spiking Neural Networks
Open accessAbstract
Event-based cameras have attracted increasing attention due to their advantages of biologically inspired paradigm and low power consumption. Since event-based cameras record the visual input as asynchronous discrete events, they are inherently suitable to cooperate with the spiking neural network (SNN). Existing works of SNNs for processing events mainly focus on the task of object recognition. However, events from the event-based camera are triggered by dynamic changes, which makes it an ideal choice to capture actions in the visual scene. Inspired by the dorsal stream in visual cortex, we propose a hierarchical SNN architecture for event-based action recognition using motion information. Motion features are extracted and utilized from events to local and finally to global perception for action recognition. To the best of the authors’ knowledge, it is the first attempt of SNN to apply motion information to event-based action recognition. We evaluate our proposed SNN on three event-based action recognition datasets, including our newly published DailyAction-DVS dataset comprising 12 actions collected under diverse recording conditions. Extensive experimental results show the effectiveness of motion information and our proposed SNN architecture for event-based action recognition.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Event-ReId-v2 demonstrates a favorable size relative to existing event-based datasets, notably n-HAR, DailyAction-DVS, and DHP19.
“Our proposed dataset demonstrates a favorable size relative to existing event-based datasets, notably in the domain of activity recognition n-HAR [16] and DailyAction-DVS [17], and human pose estimation dataset DHP19 [2] (see comparisons in Table 1).”
From Event Anonymization: Privacy-Preserving Person Re-Identification and Pose Estimation in Event-Based Vision (2024) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2007.914337 · Cited by 2,485 · see citing works
A Low Power, Fully Event-Based Gesture Recognition System
Amir, A., Taba, B., Berg, D., Melano, T., McKinstry, J., Di Nolfo, C., Nayak, T., Andreopoulos, A., Garreau, G., Mendoza, M., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.781 · Cited by 1,017 · see citing works
Posch, C., Matolin, D., & Wohlgenannt, R. (2010) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2010.2085952 · Cited by 797 · see citing works
A Biologically Inspired System for Action Recognition
Jhuang, H., Serre, T., Wolf, L., & Poggio, T. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408988 · Cited by 721 · see citing works
HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition
Lagorce, X., Orchard, G., Galluppi, F., Shi, B. E., & Benosman, R. B. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2574707 · Cited by 617 · see citing works
HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification
Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., & Benosman, R. B. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00186 · Cited by 496 · see citing works
SLAYER: Spike Layer Error Reassignment in TimePreprint
Shrestha, S. B., & Orchard, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08646 · Cited by 465 · see citing works
HFirst: A Temporal Approach to Object Recognition
Orchard, G., Meyer, C., Etienne‐Cummings, R., Posch, C., Thakor, N. V., & Benosman, R. B. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2392947 · Cited by 314 · see citing works
Asynchronous frameless event-based optical flow
Benosman, R. B., Ieng, S.-H., Clercq, C., Bartolozzi, C., & Srinivasan, M. V. (2011) · Neural Networks · doi:10.1016/j.neunet.2011.11.001 · Cited by 229 · see citing works
Paredes-Vallés, F., Scheper, K. Y. W., & de Croon, G. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2903179 · Cited by 197 · see citing works
DVS Benchmark Datasets for Object Tracking, Action Recognition, and Object Recognition
Hu, Y., Liu, H., Pfeiffer, M., & Delbrück, T. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00405 · Cited by 133 · see citing works
DART: Distribution Aware Retinal Transform for Event-based Cameras
Ramesh, B., Yang, H., Orchard, G., Thi, N. A. L., Zhang, S., & Xiang, C. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2919301 · Cited by 110 · see citing works
Xing, Y., Caterina, G. D., & Soraghan, J. J. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.590164 · Cited by 103 · see citing works
Neuromorphic Vision Datasets for Pedestrian Detection, Action Recognition, and Fall Detection
Miao, S., Chen, G., Ning, X., Zi, Y., Ren, K., Bing, Z., & Knoll, A. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00038 · Cited by 100 · see citing works
Temporal binary representation for event-based action recognition
Innocenti, S. U., Becattini, F., Pernici, F., & Bimbo, A. D. (2020) · Florence Research (University of Florence) · doi:10.1109/icpr48806.2021.9412991 · Cited by 88 · see citing works
STCA: Spatio-Temporal Credit Assignment with Delayed Feedback in Deep Spiking Neural Networks
Gu, P., Xiao, R., Pan, G., et al. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/189 · Cited by 86 · see citing works
Self-citationA spiking neural network architecture for visual motion estimation
Orchard, G., Benosman, R. B., Etienne‐Cummings, R., et al. (2013) · IEEE Biomedical Circuits and Systems Conference (BioCAS) · doi:10.1109/biocas.2013.6679698 · Cited by 70 · see citing works
Spiking Deep Residual NetworkPreprint
Hu, Y., Tang, H., & Pan, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01352 · Cited by 70 · see citing works
Self-citationMaro, J.-M., Ieng, S.-H., & Benosman, R. B. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00275 · Cited by 62 · see citing works
Liu, Q., Ruan, H., Xing, D., et al. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i02.5486 · Cited by 60 · see citing works
Self-citationAn Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning
Xiao, R., Tang, H., Ma, Y., Yan, R., & Orchard, G. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2945630 · Cited by 59 · see citing works
Self-citationUnsupervised AER Object Recognition Based on Multiscale Spatio-Temporal Features and Spiking Neurons
Liu, Q., Pan, G., Ruan, H., Xing, D., Xu, Q., & Tang, H. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2966058 · Cited by 49 · see citing works
Self-citationA Reservoir-based Convolutional Spiking Neural Network for Gesture Recognition from DVS Input
George, A. M., Banerjee, D., Dey, S., Mukherjee, A., & Balamurali, P. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206681 · Cited by 44 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Brandli et al., 2014] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240× 180 130 db 3 µs latency global shutter spatiotem- poral vision sensor. IEEE Journal of Solid-State Circuits, 49(10):2333-2341, 2014.
- Simone Undri Innocenti, Federico Becattini, Federico Pernici, and Alberto Del Bimbo. Tem- poral binary representation for event-based action recog- nition. arXiv preprint arXiv:2010.08946, 2020. · arXiv
- Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128×128 120 db 15µs la- tency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43(2):566-576, 2008.
Cited by
Sorted by citations
Person Re-Identification without Identification via Event Anonymization
Ahmad, S., Morerio, P., & Del Bue, A. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01022 · Cited by 39 · see citing works
Ahmad, S., Morerio, P., & Del Bue, A. (2024) · IEEE Access · doi:10.1109/access.2024.3399539 · Cited by 16 · see citing works
“Our proposed dataset demonstrates a favorable size relative to existing event-based datasets, notably in the domain of activity recognition n-HAR [16] and DailyAction-DVS [17], and human pose estimation dataset DHP19 [2] (see comparisons in Table 1).”
Background
- Authors
- Liu, Q., Xing, D., Tang, H., Ma, D., & Pan, G.
- Affiliations
- Zhejiang Lab (CN)
- Zhejiang University (CN)
- Venue
- Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence
- Published
- 2021-08
- Pages
- 1743-1749
- OpenAlex
- W3189897383
- Cited by
- 105
- References
- 23
- Citations by year
- 2026: 24 · 2025: 28 · 2024: 28 · 2023: 17 · 2022: 8
- FWCI
- 20.80
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- Funders
- National Natural Science Foundation of China (61925603)
- Key Research and Development Program of Zhejiang Province
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- event cameras, spiking neural networks, motion information, dorsal stream, biologically inspired vision
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
