The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception
Open accessAbstract
As a bio-inspired and emerging sensor, an event-based neuromorphic vision sensor has a different working principle compared to the standard frame-based cameras, which leads to promising properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution. It poses a paradigm shift to sense and perceive the environment by capturing local pixel-level light intensity changes and producing asynchronous event streams. Advanced technologies for the visual sensing system of autonomous vehicles from standard computer vision to event-based neuromorphic vision have been developed. In this tutorial-like article, a comprehensive review of the emerging technology is given. First, the course of the development of the neuromorphic vision sensor that is derived from the understanding of biological retina is introduced. The signal processing techniques for event noise processing and event data representation are then discussed. Next, the signal processing algorithms and applications for event-based neuromorphic vision in autonomous driving and various assistance systems are reviewed. Finally, challenges and future research directions are pointed out. It is expected that this article will serve as a starting point for new researchers and engineers in the autonomous driving field and provide a bird's-eye view to both neuromorphic vision and autonomous driving research communities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3008413 · Cited by 2,566
Self-citationA 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2007.914337 · Cited by 2,485
Autonomous driving in urban environments: Boss and the Urban Challenge
Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M. N., Dolan, J., Duggins, D., Galatali, T., Geyer, C., et al. (2008) · Journal of Field Robotics · doi:10.1002/rob.20255 · Cited by 1,555
Going Deeper in Spiking Neural Networks: VGG and Residual Architectures
Sengupta, A., Ye, Y., Wang, R., Liu, C., & Roy, K. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00095 · Cited by 1,211
Badue, C., Guidolini, R., Carneiro, R. V., Azevedo, P., Cardoso, V. B., Forechi, A., Jesus, L., Berriel, R., Paixão, T. M., Mutz, F., et al. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113816 · Cited by 1,209
Rueckauer, B., Lungu, I.-A., Hu, Y., Pfeiffer, M., & Liu, S.-C. (2017) · Frontiers in Neuroscience · doi:10.3389/fnins.2017.00682 · Cited by 1,072
Training Deep Spiking Neural Networks Using Backpropagation
Lee, J. H., Delbruck, T., & Pfeiffer, M. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00508 · Cited by 976
Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition
Cao, Y., Chen, Y., & Khosla, D. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0788-3 · Cited by 917
Deep Learning With Spiking Neurons: Opportunities and Challenges
Pfeiffer, M., & Pfeil, T. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00774 · Cited by 766
HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition
Lagorce, X., Orchard, G., Galluppi, F., Shi, B. E., & Benosman, R. B. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2574707 · Cited by 612
Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2020) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000521357 · Cited by 591
Self-citationUnsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion
Zhu, A. Z., Yuan, L., Chaney, K., & Daniilidis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00108 · Cited by 571
The Multivehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception
Zhu, A. Z., Thakur, D., Ozaslan, T., Pfrommer, B., Kumar, V., & Daniilidis, K. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2800793 · Cited by 567
Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars
Maqueda, A. I., Loquercio, A., Gallego, G., Garcia, N., & Scaramuzza, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00568 · Cited by 514
Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars
Maqueda, A. I., Loquercio, A., Gallego, G., Garcia, N. N., & Scaramuzza, D. (2018) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-175994 · Cited by 493
HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification
Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., & Benosman, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00186 · Cited by 486
Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output
Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B., & Delbruck, T. (2014) · Proceedings of the IEEE · doi:10.1109/jproc.2014.2346153 · Cited by 471
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
Zhu, A., Yuan, L., Chaney, K., & Daniilidis, K. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.062 · Cited by 470
Events-To-Video: Bringing Modern Computer Vision to Event Cameras
Rebecq, H., Ranftl, R., Koltun, V., & Scaramuzza, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00398 · Cited by 420
Benosman, R., Clercq, C., Lagorce, X., Ieng, S.-H., & Bartolozzi, C. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2273537 · Cited by 419
End-to-End Learning of Representations for Asynchronous Event-Based Data
Gehrig, D., Loquercio, A., Derpanis, K., & Scaramuzza, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00573 · Cited by 386
HFirst: A Temporal Approach to Object Recognition
Orchard, G., Meyer, C., Etienne-Cummings, R., Posch, C., Thakor, N., & Benosman, R. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2392947 · Cited by 312
Mahowald, M. A., & Mead, C. (1991) · Scientific American · doi:10.1038/scientificamerican0591-76 · Cited by 270
Continuous-Time Intensity Estimation Using Event Cameras
Scheerlinck, C., Barnes, N., & Mahony, R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20873-8_20 · Cited by 217
ESIM: an Open Event Camera Simulator
Rebecq, H., Gehrig, D., & Scaramuzza, D. (2018) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-175993 · Cited by 194
Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation
Munda, G., Reinbacher, C., & Pock, T. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1106-2 · Cited by 181
EV-SegNet: Semantic Segmentation for Event-Based Cameras
Alonso, I., & Murillo, A. C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00205 · Cited by 163
Interacting maps for fast visual interpretation
Cook, M., Gugelmann, L., Jug, F., Krautz, C., & Steger, A. (2011) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2011.6033299 · Cited by 162
EV-Gait: Event-Based Robust Gait Recognition Using Dynamic Vision Sensors
Wang, Y., Du, B., Shen, Y., Wu, K., Zhao, G., Sun, J., & Wen, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00652 · Cited by 160
Chen, N. F. Y. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00107 · Cited by 107
O(N)-Space Spatiotemporal Filter for Reducing Noise in Neuromorphic Vision Sensors
Khodamoradi, A., & Kastner, R. (2018) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2017.2788865 · Cited by 103
DDD17: End-To-End DAVIS Driving Dataset
Binas, J., Neil, D., Liu, S.-C., & Delbruck, T. (2017) · arXiv · doi:10.48550/arxiv.1711.01458 · Cited by 96
Padala, V. R., Basu, A., & Orchard, G. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00118 · Cited by 91
Chen, G., Lin, H., Dong, J., Liu, P., Conradt, J., & Knoll, A. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.2973049 · Cited by 90
Self-citationDesign of a spatiotemporal correlation filter for event-based sensors
Liu, H., Brändli, C., Li, C.-H., Liu, S., & Delbrück, T. (2015) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2015.7168735 · Cited by 89
Event-Based Vision Enhanced: A Joint Detection Framework in Autonomous Driving
Li, J., Dong, S., Yu, Z., Tian, Y., & Huang, T. (2019) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00242 · Cited by 87
Spatiotemporal multiple persons tracking using Dynamic Vision Sensor
Piatkowska, E., Belbachir, A., Schraml, S., & Gelautz, M. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops · doi:10.1109/cvprw.2012.6238892 · Cited by 82
DET: A High-Resolution DVS Dataset for Lane Extraction
Cheng, W., Luo, H., Yang, W. T., Yu, L., Chen, S., & Li, W. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00210 · Cited by 81
Event-Driven Sensing for Efficient Perception: Vision and Audition Algorithms
Liu, S., Rueckauer, B., Ceolini, E., Huber, A. E. G., & Delbrück, T. (2019) · IEEE Signal Processing Magazine · doi:10.1109/msp.2019.2928127 · Cited by 77
An Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning
Xiao, R., Tang, H., Ma, Y., Yan, R., & Orchard, G. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2945630 · Cited by 62
Self-citationInceptive Event Time-Surfaces for Object Classification Using Neuromorphic Cameras
Baldwin, R. W., Almatrafi, M., Kaufman, J. R., Asari, V. K., & Hirakawa, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-27272-2_35 · Cited by 58
Spike Coding for Dynamic Vision Sensor in Intelligent Driving
Dong, S., Bi, Z., Tian, Y., & Huang, T. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2872984 · Cited by 47
Event-driven stereo matching for real-time 3D panoramic vision
Schraml, S., Belbachir, A. N., & Bischof, H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298644 · Cited by 44
Multi-Cue Event Information Fusion for Pedestrian Detection With Neuromorphic Vision Sensors
Chen, G., Cao, H., Ye, C., Zhang, Z., Liu, X., Mo, X., Qu, Z., Conradt, J., Röhrbein, F., & Knoll, A. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00010 · Cited by 44
Self-citationRobust Event-Based Object Tracking Combining Correlation Filter and CNN Representation
Li, H., & Shi, L. (2019) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2019.00082 · Cited by 40
Folowosele, F. O., Vogelstein, R. J., & Etienne‐Cummings, R. (2011) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2012.2183409 · Cited by 22
Chen, G., Chen, J., Lienen, M., Conradt, J., Röhrbein, F., & Knoll, A. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00073 · Cited by 19
Self-citationCross-Modal Learning Filters for RGB-Neuromorphic Wormhole Learning
Zanardi, A., Aumiller, A. J., Zilly, J., Censi, A., & Frazzoli, E. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.045 · Cited by 14
Spiking Neural Network Based Region Proposal Networks for Neuromorphic Vision Sensors
Acharya, J., Padala, V. R., & Basu, A. (2019) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2019.8702651 · Cited by 7
Zanardi, A., Zilly, J., Aumiller, A. J., Censi, A., & Frazzoli, E. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794336 · Cited by 4
Cited by
Chen, G., Wang, H., Chen, K., Li, Z., Song, Z., Liu, Y., Chen, W., & Knoll, A. (2022) · IEEE Transactions on Systems, Man, and Cybernetics: Systems · doi:10.1109/tsmc.2020.3005231 · Cited by 354
Self-citationComputational event-driven vision sensors for in-sensor spiking neural networks
Zhou, Y., Fu, J., Chen, Z., Zhuge, F., Wang, Y., Yan, J., Ma, S., Xu, L., Yuan, H., Chan, M., et al. (2023) · Nature Electronics · doi:10.1038/s41928-023-01055-2 · Cited by 283
YOLOv5-Tassel: Detecting Tassels in RGB UAV Imagery With Improved YOLOv5 Based on Transfer Learning
Liu, W., Quijano, K., & Crawford, M. M. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3206399 · Cited by 270
Automated Vehicle Sideslip Angle Estimation Considering Signal Measurement Characteristic
Liu, W., Xia, X., Xiong, L., Lu, Y., Gao, L., & Yu, Z. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3059050 · Cited by 165
Metadata
- Authors
- Chen, G., Cao, H., Conradt, J., Tang, H., Rohrbein, F., & Knoll, A.
- Venue
- IEEE Signal Processing Magazine
- Published
- 2020-07
- Volume
- 37
- Issue
- 4
- Pages
- 34-49
- ISSN
- 1053-5888 (ISSN-L)
- 1558-0792
- OpenAlex
- W3039515597
- Cited by
- 304
- References
- 50
- FWCI
- 14.70
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.