N-HAR: A Neuromorphic Event-Based Human Activity Recognition System using Memory Surfaces
Abstract
In recent years, a new generation of low-power, neuromorphic, event-based vision sensors has been gaining popularity for their very low latency and data sparsity. Though the conventional frame-based cameras have advanced in a lot of ways, they suffer from data redundancy and temporal latency. The bio-inspired artificial retinas eliminate the data redundancy by capturing only the change in illumination at each pixel and asynchronously communicating in binary spikes. In this work, we propose a system to achieve the task of human activity recognition based on the event-based camera data. We show that such tasks, which generally need high frame rate sensors for accurate predictions, can be achieved by adapting existing computer vision techniques to the spiking domain. We used event memory surfaces to make the sparse event data compatible with deep convolutional neural networks (CNNs). We leverage upon the recent advances in deep convolutional networks based video analysis and adapt such frameworks onto the neuromorphic domain. We also provide the community with a new dataset consisting of five categories of human activities captured in real world without any simulations. We achieved an accuracy of 94.3% using event memory surfaces on our activity recognition dataset.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Event-ReId-v2 demonstrates a favorable size relative to existing event-based datasets, notably n-HAR, DailyAction-DVS, and DHP19.
“Our proposed dataset demonstrates a favorable size relative to existing event-based datasets, notably in the domain of activity recognition n-HAR [16] and DailyAction-DVS [17], and human pose estimation dataset DHP19 [2] (see comparisons in Table 1).”
From Event Anonymization: Privacy-Preserving Person Re-Identification and Pose Estimation in Event-Based Vision (2024) · cites this paper as [16]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,369 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · see citing works
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_2 · Cited by 4,164 · see citing works
Recognizing human actions: a local SVM approach
Schüldt, C., Laptev, I., & Caputo, B. (2004) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2004.1334462 · Cited by 3,022 · see citing works
Recognizing human actions: A local SVM approach
Schüldt, C., Laptev, I., & Caputo, B. (2004) · Cited by 2,927 · see citing works
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A. J., & Zisserman, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.213 · Cited by 2,842 · see citing works
A 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2007.914337 · Cited by 2,485 · see citing works
Keras: The Python Deep Learning library
Chollet, F. (2018) · Astrophysics Source Code Library · Cited by 1,002 · see citing works
Posch, C., Matolin, D., & Wohlgenannt, R. (2010) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2010.2085952 · Cited by 797 · see citing works
Point-to-point connectivity between neuromorphic chips using address events
Boahen, K. A. (2000) · IEEE Transactions on Circuits and Systems II Analog and Digital Signal Processing · doi:10.1109/82.842110 · Cited by 656 · see citing works
HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition
Lagorce, X., Orchard, G., Galluppi, F., Shi, B. E., & Benosman, R. B. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2574707 · Cited by 617 · see citing works
Beyond Frame-level CNN: Saliency-Aware 3-D CNN With LSTM for Video Action Recognition
Wang, X., Gao, L., Song, J., & Shen, H. T. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2611485 · Cited by 264 · see citing works
Asynchronous frameless event-based optical flow
Benosman, R. B., Ieng, S.-H., Clercq, C., Bartolozzi, C., & Srinivasan, M. V. (2011) · Neural Networks · doi:10.1016/j.neunet.2011.11.001 · Cited by 229 · see citing works
Asynchronous Event-Based Multikernel Algorithm for High-Speed Visual Features Tracking
Lagorce, X., Meyer, C., Ieng, S.-H., Filliat, D., & Benosman, R. B. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2352401 · Cited by 166 · see citing works
DVS Benchmark Datasets for Object Tracking, Action Recognition, and Object Recognition
Hu, Y., Liu, H., Pfeiffer, M., & Delbrück, T. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00405 · Cited by 133 · see citing works
Event-based Sensing for Space Situational Awareness
Cohen, G., Afshar, S., Morreale, B., Bessell, T., Wabnitz, A., Rutten, M. G., & André, V. S. (2019) · The Journal of the Astronautical Sciences · doi:10.1007/s40295-018-00140-5 · Cited by 92 · see citing works
Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas
Lee, J. H., Delbrück, T., Pfeiffer, M., Park, P. K. J., Shin, C.-W., Ryu, H., & Kang, B. C. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2308551 · Cited by 84 · see citing works
Spatiotemporal features for asynchronous event-based data
Lagorce, X., Ieng, S.-H., Clady, X., Pfeiffer, M., & Benosman, R. B. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00046 · Cited by 53 · see citing works
Real time unsupervised learning of visual stimuli in neuromorphic VLSI systems
Giulioni, M., Corradi, F., Dante, V., & Giudice, P. D. (2015) · Scientific Reports · doi:10.1038/srep14730 · Cited by 31 · see citing works
Event-based Sensing for Space Situational Awareness
Cohen, G., Afshar, S., van Schaik, A., et al. (2017) · Cited by 14 · see citing works
Can Silicon Retina Sensors be used for optical motion analysis in sports?
Litzenberger, S., & Sabo, A. (2012) · Procedia Engineering · doi:10.1016/j.proeng.2012.04.128 · Cited by 5 · see citing works
Afshar, S., Cohen, G., Hamilton, T. J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.04223 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Person Re-Identification without Identification via Event Anonymization
Ahmad, S., Morerio, P., & Del Bue, A. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01022 · Cited by 39 · see citing works
Event Encryption for Neuromorphic Vision Sensors: Framework, Algorithm, and Evaluation
Du, B., Li, W., Wang, Z., Xu, M., Gao, T., Li, J., & Wen, H. (2021) · Sensors · doi:10.3390/s21134320 · Cited by 18 · see citing works
Ahmad, S., Morerio, P., & Del Bue, A. (2024) · IEEE Access · doi:10.1109/access.2024.3399539 · Cited by 16 · see citing works
- Authors
- Pradhan, B. R., Bethi, Y., Narayanan, S., Chakraborty, A., & Thakur, C. S.
- Affiliations
- Indian Institute of Science Bangalore (IN)
- Venue
- 2019 IEEE International Symposium on Circuits and Systems (ISCAS)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 1-5
- ISSN
- 0271-4302 (ISSN-L)
- OpenAlex
- W2942896434
- Cited by
- 31
- References
- 28
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 8 · 2023: 3 · 2022: 4 · 2021: 2 · 2020: 6
- FWCI
- 4.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE International Symposium on Circuits and Systems proceedings (publishedVersion)
- NOT FOUND REPOSITORY (Indian Institute of Science Bangalore) (acceptedVersion)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Memory and Neural Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- human activity recognition, event cameras, spiking neural networks, event-based vision, convolutional neural networks, activity recognition dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
